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PREFACE 


In  its  performance-based  services  acquisition  (PBSA)  activities,  the  Air  Force  focuses  on 
telling  a  provider  what  the  Air  Force  needs  rather  than  how  the  provider  should  meet  that 
need.  The  Office  of  the  Secretary  of  Defense  has  set  an  aggressive  policy  that  seeks  to 
implement  such  services  acquisition  on  a  broad  scale.  The  Air  Force  is  pursuing  an  even 
more  aggressive  implementation  program.  This  documented  briefing  seeks  to  support  the 
continuing  implementation  of  PBSA  by  identifying  Air  Force  bases  that  have  succeeded 
in  implementing  it  in  specific  contracts,  describing  details  of  how  they  did  this,  and  then 
developing  lessons  learned  that  the  remainder  of  the  Air  Force  can  use  to  build  on 
progress  made  to  date. 

The  work  draws  on  detailed,  structured  interviews  with  relevant  contracting  and 
functional  personnel  at  15  Air  Force  bases  responsible  for  22  performance-based 
contracts  awarded  between  FY  1998  and  FY  2000.  The  contracts  focus  on  the  activities 
of  greatest  day-to-day  importance  to  operational  contracting:  military  family  housing 
maintenance,  grounds  maintenance,  custodial  services,  food  services,  and  refuse  services. 

This  research  was  sponsored  by  the  Office  of  the  Deputy  Assistant  Secretary  of  the  Air 
Force  (Contracting),  SAF/AQC.  Although  the  analysis  focuses  on  recent  Air  Force 
experience,  it  should  interest  anyone  concerned  with  the  implementation  of  PBSA  in  a 
government  setting,  particularly  its  application  to  the  kinds  of  base  services  identified 
above.  The  research  was  completed  in  January  2001. 

Research  on  this  and  related  contracting  topics  continues  in  the  Resource  Management 
Program  of  Project  AIR  FORCE.  For  additional  information,  please  contact  the  study 
leader.  Dr.  Laura  Baldwin,  at  Laura_Baldwin@raod.org.  To  convey  conunents  on  this 
document,  please  contact  the  lead  author,  John  Ausink,  at  John_Ausink@rand.org  or 
(703)  413-1 100,  extension  5403. 


PROJECT  Am  FORCE 

Project  AIR  FORCE,  a  division  of  RAND,  is  the  Air  Force  federally  funded  research  and 
development  center  (FFRDC)  for  studies  and  analyses.  It  provides  the  Air  Force  with 
independent  analyses  of  policy  alternatives  affecting  the  development,  employment, 
combat  readiness,  and  support  of  current  and  future  aerospace  forces.  Research  is 
performed  in  four  programs:  Aerospace  Force  Development;  Manpower,  Personnel,  and 
Training;  Resource  Management;  and  Strategy  and  Doctrine. 
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SUMMARY 


A  performance-based  contract  tells  the  seller  what  the  buyer  wants  done,  not  how  to  do  it, 
and  since  1991  it  has  been  the  policy  of  the  federal  government  that  agencies  use 
performance-based  contracting  methods  to  the  maximum  extent  practicable  when 
acquiring  services.  Department  of  Defense  interest  in  performance-based  contracting  has 
increased  in  recent  years,  and  in  April  2000  the  Under  Secretary  of  Defense  for 
Acquisition  and  Technology  directed  that  50  percent  of  service  acquisitions  be 
performance  based  by  the  year  2005.  A  variety  of  Air  Force  instructions,  publications, 
and  policies  are  in  place  to  aggressively  implement  this  policy. 

The  primary  goal  of  this  study  is  to  use  examples  of  successful  Air  Force  applications  of 
performance-based  services  acquisition  (PBSA)  to  illustrate  how  to  pursue  these  practices 
elsewhere.  In  this  documented  briefing,  the  results  of  interviews  with  personnel  at  Air 
Force  bases  that  have  used  PBSA  are  arranged  as  a  'Svalk-through”  of  the  process  of 
applying  these  techniques  to  the  development  of  a  contract.  On  the  basis  of  these 
interview  results,  we  reach  several  conclusions  about  what  seems  to  work  in  the  field  and 
what  needs  to  be  improved,  and  we  then  summarize  these  conclusions  in  a  list  of  lessons 
learned  from  experiences  in  PBSA  implementation. 


INTERVIEW  RESULTS 

We  conducted  interviews  with  contracting  and  functional  (e.g.,  civil  engineering  [CE] 
and  services  [SV])  personnel  at  15  Air  Force  bases  representing  five  major  commands 
(MAJCOMs).  In  these  interviews,  we  discussed  the  implementation  of  PBSA  in  22 
recent  contracts — most  of  them  for  militaiy  family  housing  (MFH)  maintenance,  grounds 
maintenance,  custodial  services,  or  food  services.  The  15  bases  were  ‘‘self-selected”  in 
that  they  had  responded  to  a  request  for  examples  of  “successful”  implementation  of 
PBSA  practices  and  had  done  so  on  the  basis  of  their  own  perception  of  what  success 
means. 

Motivation  for  Using  PBSA 

Most  bases  applied  PBSA  techniques  because  of  MAJCOM  direction  or  the  introduction 
of  Air  Force  Instruction  (API)  63-124.^  A  few,  however,  used  the  approach  because  of 
dissatisfaction  with  the  performance  of  contractors  selected  by  other  methods  and  would 
thus  have  used  PBSA  even  without  headquarters  direction.  These  self-motivated 
contracting  offices  generally  expressed  the  most  satisfaction  with  the  outcome  of  the 
approach. 


^AFI 63-124,  Performance-Based  Service  Contracts  (PBSC),  April  1, 1999.  This  instruction  prescribes 
how  the  Air  Force  will  implement  performance-based  services  contracting  and  quality  assurance  in 
accordance  with  Federal  Acquisition  Regulations. 


Getting  Started 

AFI  63-124  directs  the  establishment  of  a  customer-focused,  multifunctional  team  called 
a  Business  Requirements  Advisory  Group  (BRAG)  to  manage  a  service  contract 
throughout  the  life  of  a  requirement,  and  a  large  majority  of  the  contracts  in  this  study 
(some  of  them  developed  before  the  release  of  AFI  63-124)  used  formal  BRAGs  or 
equivalent  teams.  There  was  genuine  enthusiasm  for  the  effectiveness  of  these  teams; 
they  were  said  not  only  to  have  increased  the  efficiency  of  the  process  of  contract 
development,  but  also  to  have  helped  team  members  develop  a  sense  of  collective 
“ownership”  of  the  process. 

Training 

The  contracts  in  this  study  were  awarded  from  FY  1998  to  FY  2000,  with  the  most  recent 
having  been  awarded  in  mid-FY  2000.  Everyone  we  interviewed  said  that  they  had 
received  insufficient  training  before  being  asked  to  implement  PBSA  practices.  Although 
many  had  received  basic  briefings  on  different  aspects  of  PBSA  (writing  statements  of 
work  [SOWs],  for  example)  and  some  had  received  excellent  assistance  from  MAJCOM 
headquarters,  the  majority  opinion  was  that  more  training  was  needed  earlier  in  the 
process.  Most  of  those  interviewed  expressed  a  desire  for  “hands-on”  training  with 
detailed  examples  and  step-by-step  work  through  the  development  of  a  PBSA  contract. 
SAF/AQC  has  begun  efforts  to  improve  training  (and  some  new  training  programs  were 
praised  by  the  one  person  we  interviewed  who  had  recently  attended),  but  those  we 
interviewed  also  noted  that  basic  training  in  the  contracting  career  field  does  not  yet 
address  PBSA  practices.  Several  interviewees  also  mentioned  that  training  should  be 
provided  to  functional  representatives  in  addition  to  contracting  personnel. 

Market  Research 

Personnel  we  interviewed  used  a  variety  of  methods  in  conducting  market  research, 
ranging  from  combing  the  Yellow  Pages  to  conducting  site  visits  at  organizations  that 
were  using  the  services  of  interest.  Although  some  found  it  difficult  to  find  commercial 
firms  that  operated  imder  circumstances  similar  enough  to  those  of  the  Air  Force  to 
provide  useful  information,  many  were  able  to  use  their  market  research  to  aid  in  the 
development  of  the  SOW  and  to  provide  insight  into  performance  factors  that  should  be 
considered  in  the  solicitation. 

Defining  Requirements 

In  conferences  and  seminars.  Air  Force  MAJCOM  representatives  have  speculated  that 
implementing  PBSA  practices  would  require  waivers  of  many  AFIs  and  the  eventual 
revision  of  others.  Because  of  these  comments,  it  was  surprising  that,  with  only  one 
exception,  no  mention  was  made  in  our  interviews  of  formal  MAJCOM  guidance  for 
PBSA  implementation,  and  no  bases  revealed  that  they  had  to  request  waivers  for  any 
AFIs.  At  some  bases,  frustration  was  expressed  that  they  would  write  SOWs  in 
accordance  with  perceived  MAJCOM  guidance  only  to  be  directed  to  change  what  they 
had  done  because  of  changes  in  headquarters  policy  that  had  not  been  annoimced. 

Air  Education  and  Training  Command  (AETC)  appears  to  have  fairly  strict  guidelines  for 
what  level  of  service  is  allowable  in  grounds  maintenance,  custodial,  and  refuse/recycling 
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contracts.  Bases  that  desire  higher  levels  of  service  must  pay  for  them  by  diverting  funds 
from  other  sources.  Although  none  of  the  AETC  personnel  we  interviewed  said  this  was 
a  problem,  it  is  easy  to  imagine  scenarios  in  which  the  Air  Force  might  not  get  “best 
value”  if  a  base  were  faced  with  the  choice  of  either  paying  out  of  its  own  pocket  to  get 
higher  performance  or  settling  on  a  poorer  performer  that  just  met  the  AETC  standard. 

SOW  and  Quality  Assurance  Surveillance  Plan  Development 

The  majority  of  the  personnel  we  interviewed  reported  substantial  decreases  in  the 
lengths  of  SOWs  for  their  contracts,  and  review  of  the  actual  SOWs  showed  that  many  of 
these  decreases  had  resulted  from  language  changes  that  focused  on  what  was  to  be 
accomplished  rather  than  how  to  accomplish  it.  In  only  one  case  did  it  appear  that  the 
SOW  was  shorter  because  it  made  references  to  APIs  rather  than  quoting  from  them. 

All  the  bases  in  the  study  reported  a  shift  to  customer  complaint  as  the  primary  method  of 
quality  assurance,  although  some  had  implemented  the  approach  more  thoroughly  than 
others.  In  fact,  some  quality  assurance  evaluators  who  claimed  to  rely  on  customer 
complaints  still  conducted  100  percent  inspections.^  Several  people  noted  that  the 
dependence  on,  and  the  level  of^  inspections  rested  on  the  nature  of  the  service  and 
vendor  expertise  and  would  be  adjusted  in  the  future  if  performance  so  warranted. 
Customer  education  about  what  constituted  a  valid  complaint  was  also  judged  to  be 
important. 

An  important  and  frequent  comment  from  the  field  was  that  the  shift  to  a  customer 
complaint  environment  had  contributed  to  a  partnering  rather  than  an  adversarial 
relationship. 

Contract  Structure 

All  but  three  of  the  contracts  in  this  study  were  firm  fixed  price  (FFP).  With  few 
exceptions,  these  contracts  were  for  terms  of  a  basic  period  plus  four  option  years. 

Award  fees  were  the  exception;  only  six  of  the  contracts  had  them.  However,  reasons  for 
not  having  award  fees  varied:  Some  bases  did  not  have  the  money  to  provide  them;  some 
felt  that  such  fees  were  inappropriate  (in  custodial  contracts,  for  example,  how  can 
performance  merit  a  fee  by  being  “cleaner  than  clean”?);  and  others  felt  that  the  Air 
Force’s  use  of  past-performance  information  was  sufficient  motivation  for  a  contractor  to 
do  a  good job. 

Most  contracting  officers  said  that  they  had  not  considered  using  an  award  term  as  an 
incentive  in  a  contract,  but  one  Air  Combat  Command  (ACC)  base  was  experimenting 
with  the  approach  in  an  MFH  maintenance  contract. 

Source  Selection 

The  most  popular  forms  of  source  selection  were  “best  value”  (9  of  the  22  contracts)  and 
the  performance-price  tradeoff  (PPT)  method  (10  of  the  22).  A  best  value  approach 
allows  for  the  consideration  of  factors  other  than  price  in  awarding  a  contract  (e.g.,  past 
performance  of  the  vendor  and  technical  expertise)  so  that  the  contract  may  not 
necessarily  be  awarded  to  the  lowest  bidder.  The  PPT  method  is  a  simplified  best  value 


^That  is,  perfonnance  is  inspected  or  evaluated  each  time  a  task  is  accomplidied. 
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approach  that  limits  the  evaluation  factors  to  price  and  past  performance.  Air  Mobility 
Command  (AMC)  appears  to  be  encouraging  the  PPT  approach  more  actively  than  the 
other  commands. 

Past  Performance 

All  the  personnel  we  interviewed  were  enthusiastic  about  the  use  of  past-performance 
information  in  the  source  selection  process.  Some  used  “standard”  questionnaires  to 
gather  the  information  but  noted  that  in  the  future  they  would  work  to  customize  past- 
performance  questionnaires  to  their  specific  needs.  Most  used  follow-up  telephone  calls 
to  clarify  questions  about  past  performance  as  well  as  informal  Air  Force  contacts  to 
gather  more  candid  information.  Several  commented  that  it  might  be  useful  to  develop  a 
central  database  of  past-performance  information  on  contractors  that  have  worked  with 
the  Air  Force. 


RESULTS  OF  PBSA  IMPLEMENTATION 
Effects  on  Performance 

Those  we  interviewed  were  very  happy  with  the  performance  of  the  vendors  they  had 
obtained  using  PBSA  practices.  However,  not  enough  time  has  elapsed  to  allow  for  an 
objective  evaluation  of  changes  in  performance  because  most  of  the  contracts  we 
discussed  were  relatively  young — i.e.,  in  force  for  approximately  six  months. 
Significantly,  we  saw  no  indication  that  base  officials  were  formally  comparing  the 
performance  of  the  current  contractor  with  that  of  the  previous  contractor.  For  example, 
some  officials  told  us  that  complaints  were  down  under  the  new  contractor,  but  they  had 
not  performed  any  analysis  of  the  record  of  complaints  for  the  previous  contractor.  Other 
bases  required  that  the  contractor  keep  metrics  that  could  be  reviewed  by  the  base  and 
thus  relied  on  the  contractor  to  track  trends.  Although  this  is  an  effective  approach 
toward  managing  the  current  contract,  it  still  does  not  help  the  Air  Force  measure 
improvement  over  previous  contracts. 

Surprisingly,  at  only  two  bases  did  personnel  attribute  their  success  in  obtaining  better 
contractor  performance  to  the  use  of  new  acquisition  practices  and  to  the  care  they  might 
have  taken  in  reviewing  past  performance.  Most  commented  that  they  were  “lucky”  to 
have  found  a  good  contractor. 

Effects  on  Costs 

Changes  in  costs  resulting  fi'om  PBSA  practices  were  also  difficult  to  determine  because 
of  the  youth  of  the  contracts,  but  three  other  factors  made  the  determination  even  more 
complex.  First,  many  of  the  new  contracts  are  hard  to  compare  to  old  contracts  because 
the  scope  of  work  has  changed.  Second,  it  is  difficult  to  determine  the  accuracy  of 
government  cost  estimates  that  are  a  standard  part  of  an  acquisition.  Third,  it  is  difficult 
to  measure  changes  in  “internal”  costs — i.e.,  those  incurred  in  the  preparation  of  a 
contract  using  new  methods  and  those  incurred  in  the  management  of  a  contract  once  it  is 
complete.  For  example,  there  were  obvious  costs  related  to  increased  time  spent  on 
market  research  and  learning  how  to  write  a  performance-based  SOW.  Yet  while  many 


XI 


of  those  interviewed  alluded  to  these  costs,  their  sense  of  them  was  highly 
impressionistic. 

With  respect  to  Air  Force  performance  management  costs,  we  expected  that  the  new 
focus  on  customer  complaints  would  lead  to  a  reduction  in  quality  assurance  evaluator 
(QAE)  manpower  or  workload  and  hence  to  a  significant  reduction  in  costs.  But  the 
comments  from  our  interviews  are  ambiguous.  Some  bases  found  QAE  workload  to  be 
increasing  because  of  the  expertise  required  to  write  new  contracts  using  PBSA  practices; 
others  reported  that  they  still  have  the  same  number  of  QAEs  because  they  continue  to  do 
100  percent  inspections  as  they  adjust  to  the  use  of  customer  complaints.  Those  who 
expected  to  transition  from  inspections  to  customer  complaints  as  vendor  performance 
warranted  were  unsure  of  the  effect  this  would  have  on  future  demand  for  QAEs. 

Attitudes  Toward  PBSA  Practices 

Customers,  contracting  officers,  and  vendors  were  generally  satisfied  with  PBSA 
approaches  to  contracting.  Those  we  interviewed  said  that  customer  complaints  were 
down;  contracting  officers  felt  that  the  process  was  working  as  it  should;  and  vendors 
appreciated  being  treated  as  part  of  a  team.  The  attitudes  of  functionals,  however,  were 
mixed.  Some  favored  the  team  environment  encouraged  by  the  cross-functional  approach 
prescribed  in  AFI 63-124,  but  others  were  skeptical  about  performance-based  SOWs  that 
they  felt  gave  them  less  control  over  the  vendor. 

Reactions  to  AFI  63-124  were  also  mixed.  Some  felt  that  the  flexibility  of  this  document 
gave  them  an  opportunity  to  improve  contracts;  others  felt  that  its  ambiguity  forced  them 
to  rely  too  much  on  other  documents  for  guidance. 


CONCLUSIONS 
Practices  That  Seem  to  Work 

Teamwork,  market  research,  and  the  use  of  past-performance  information  are  three 
practices  that  appear  to  be  working  well  in  the  field.  Teamwork  helps  encourage  “buy- 
in”  of  PBSA  practices  by  educating  all  participants  about  them  and  keeping  everyone 
informed  during  the  process  of  contract  development.  In  the  process  of  developing  the 
SOW,  teaming  also  helps  the  contracting  office  understand  the  real  needs  of  the 
functional  customer  while  helping  the  functional  customer  understand  how  the 
contracting  office  needs  to  structure  the  contract  to  make  it  performance  based.  Those 
who  believed  that  working  together  helped  meet  the  needs  of  the  functional  customer  and 
give  the  Air  Force  better  value  were  most  pleased  with  the  PBSA  approach. 

Market  research  is  regarded  positively  by  those  we  talked  to  in  the  field,  and  they  seem 
pleased  with  the  way  it  helped  them  apply  commercial  standards  to  their  SOWs. 

Eveiyone  was  pleased  with  the  use  of  past-performance  information  in  evaluating 
offerors  despite  the  added  workload  it  required.  They  asserted  that  the  Air  Force  gets 
better  value,  and  the  knowledge  that  past-performance  information  is  used  provides 
incentive  for  vendors  to  perform  well  now  in  order  to  get  new  business  in  the  future. 
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What  Needs  to  Be  Improved? 

The  first  thing  that  needs  to  be  improved  is  Air  Force  training  in  PBSA  practices.  Our 
interviews  indicated  that  what  is  desired  is  training  that  provides 

•  A  better  understanding  of  how  commercial  firms  do  things. 

•  A  hands-on  explanation  of  how  PBSA  works  in  the  Air  Force  setting,  preferably 
illustrated  with  suitable  case  studies.  The  closer  the  case  studies  are  to  the  services  a 
base  is  about  to  buy,  the  better. 

•  A  better  understanding  of  where  to  turn  to  get  the  best  and  most  up-to-date 
information  on  PBSA  in  the  Air  Force. 

A  second  need  in  the  field  is  better  and  more  consistent  guidance  on  the  application  of 
PBSA  in  the  Air  Force  and  better  examples  in  AFIs  that  discuss  the  approach. 

Finally,  more  attention  needs  to  be  paid  to  the  collection  of  data  on  how  PBSA  is 
changing  performance  and  costs  associated  with  purchased  services.  The  adoption  of 
performance-based  contracting  methods  is  ultimately  a  way  to  help  the  Air  Force 
accomplish  its  mission  more  efficiently  and  to  obtain  the  best  value  for  the  American 
taxpayer,  and  the  support  of  leadership  at  the  base  level  is  critical  to  the  successful 
adoption  of  PBSA  methods.  Without  data  to  show  that  PBSA  is  helping  them  do  their 
job  better,  however,  these  leaders  will  be  hard  to  win  over.  With  good  data,  the  Air  Force 
will  be  able  to  continually  improve  the  application  of  PBSA  as  it  learns  from 
accumulating  experience. 


LESSONS  LEARNED 

Appendix  A,  a  key  part  of  this  document,  synthesizes  varied  comments  from  our 
interviews  into  a  set  of  lessons  learned.  We  hope  these  lessons  will  be  useful  to  those  in 
the  field  who  have  not  yet  had  the  opportunity  to  apply  PBSA  practices. 
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A  Report  on  Experiences  in  the  Fieid 
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INTRODUCTION 

The  Office  of  the  Secretary  of  Defense  has  established  an  aggressive  policy  that  seeks  to 
implement  performance-based  services  acquisition  (PBSA)  on  a  broad  scale.  The  Air 
Force  is  pursuing  an  even  more  aggressive  implementation  program.  This  document 
seeks  to  support  continuing  implementation  by  identifying  Air  Force  bases  that  have 
succeeded  in  implementing  PBSA  in  specific  contracts,  describing  the  details  of  how  they 
did  this,  and  then  developing  lessons  learned  that  the  remainder  of  the  Air  Force  can  use 
to  build  on  progress  made  to  date. 

The  work  draws  on  detailed,  structured  interviews  with  relevant  contracting  and 
functional  personnel  at  15  bases  responsible  for  22  performance-based  contracts  awarded 
between  FY  1998  and  FY  2000.  The  contracts  focus  on  the  activities  of  greatest  day-to- 
day  importance  to  operational  contracting:  military  family  housing  (MFH)  maintenance, 
grounds  maintenance,  custodial  services,  food  services,  and  refuse  services. 


2 


What  a  Performance-Based  Contract  Is 

•  Describes  requirements  in  terms  of  results 
required  rather  than  methods  of  performance 

•  Uses  measurable  performance  standards  (i.e., 
terms  of  quality,  timeliness,  quantity,  etc.)  and 
quality  assurance  surveillance  plans 

•  Uses  procedures  to  reduce  price  when 
services  are  not  performed  or  do  not  meet 
contract  requirements 

«  Uses  performance  incentives  where 
appropriate 
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The  use  of  performance-based  services  contracts  is  fairly  new  to  the  Department  of 
Defense  (DoD),  but  Federal  Acquisition  Regulations  (FARs)  have  recognized  this  form 
of  contracting  for  a  long  time.  FAR  Part  37  defines  a  performance-based  contract  in  the 
following  terms: 

•  The  buyer  tells  the  seller  what  the  buyer  wants  done,  not  how  to  do  it.  This  is  often  a 
subtle  distinction,  but  it  points  to  an  important  division  of  labor.  The  buyer  focuses  on 
translating  its  strategic  needs  into  a  clear  statement  of  how  each  service  it  buys  relates 
to  those  strategic  needs.  The  seller  focuses  in  turn  on  finding  the  best  way  to  fulfill 
the  buyer’s  needs  for  each  service  in  question.  Another  way  to  make  this  distinction  is 
the  following:  The  buyer  steers;  the  seller  rows.^ 

•  The  buyer  states  what  it  wants  in  clear  terms  that  make  measurement  as  simple  as 
possible.  The  more  objective  and  quantitative  the  terms  of  the  agreement,  the  better. 
That  said,  softer,  more  qualitative  terms  should  not  be  neglected  if  they  are  important 
to  performance.  The  buyer  should,  however,  seek  to  state  these  terms  in  simple  ways 
that  will  not  lead  to  confusion.  Once  the  terms  are  stated,  a  performance-based 
contract  calls  for  a  formal  quality  assurance  program  to  monitor  these  terms  on  an 
ongoing  basis. 

•  The  buyer  links  the  terms  of  the  agreement  to  formal  incentives.  The  FAR  emphasizes 
penalties  for  nonperformance  that  take  the  form  of  deductions  from  the  price  set  for 
the  service.  However,  DoD  has  agreed  with  the  Air  Force  that  reperformance — 
timely,  satisfactory  completion  of  a  task  at  the  provider’s  expense — is  an  acceptable 
substitute  for  deductions.  Where  appropriate,  the  buyer  can  use  positive  incentives 
such  as  an  award  fee  or  a  contract  extension  (award  term)  to  reward  a  seller  for 
excellent  service. 


^David  Osborne  and  Ted  Gaebler,  Reinventing  Government^  Reading,  MA:  Addison- Wesley,  1992. 
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PBSA  takes  a  broader  view  of  the  cross-functional  context  in  which  the  Air  Force  makes 
use  of  performance-based  services  contracts.  This  broader  view  is  still  fairly  new  to  the 
Air  Force.^  Our  analysis  drove  home  the  point  that  the  contracting  office  cannot  create 
successful  performance-based  services  contracts  by  itself.  Ultimate  customers,  relevant 
functions,  and  other  support  activities  must  get  involved  to  ensure  success.^  In  this 
presentation,  we  always  take  this  broader  perspective. 


^PBSA  and  performance-based  services  contracting  (PBSC)  are  terms  that  are  used  interchangeably  in  Air 
Force  documents  to  refer  to  the  practice  of  writing  acquisition  contracts  that  meet  the  FAR  Part  37  criteria 
for  a  performance-based  contract.  Air  Force  trainmg  materials  for  PBSA,  however,  include  discussions  of 
broader  topics,  including  training,  market  research,  and  best  value  source  selection.  We  could  say  that 
PBSC  is  about  writing  a  performance-based  contract;  v\^ereas  PBSA  is  about  acquiring  services  with  a 
performance-based  contract.  PBSA  requires  the  active  participation  of  many  professionals  outside  the 
contracting  community  and  includes  some  practices  in  addition  to  writing  a  performance-based  contract. 
^Commercial-style  PBSA  always  distinguishes  “customers”  from  “functionals.”  In  the  Air  Force  context, 
commercial  users  of  PBSA  would  refer  to  war  fighters  and  military  families  as  the  ultimate  customers  and 
to  organi2ations  like  civil  engineering  (CE)  and  services  (SV)  as  functionals.  PBSA  in  the  Air  Force  does 
not  emphasize  such  a  distinction.  In  Air  Force  PBSA,  CE  and  SV  can  be  treated  as  the  customers  ^\blO 
determine  ^\Ilether  a  contractor  is  performing  appropriately.  This  document  uses  customers  in  both  ways, 
relying  on  the  context  to  make  the  distinction  clear.  If  uncertain,  assume  that  the  Air  Force  view  holds. 
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Overview 

Background 

-  Government  guidance  on  PBSA 

-  Air  Force  emphasis 
<  Goal  of  this  study 

-  Focus  areas  and  interview  sources 

•  Interview  results 

>  Walking  through  the  process 

-  Results  of  PBSA  Implementation 

•  Conclusions 
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BACKGROUND 

This  documented  briefing  focuses  on  recent  Air  Force  experience  with  PBSA.  It  starts 
with  a  brief  discussion  of  background  information  relevant  to  this  experience,  but  the 
body  of  the  briefing  focuses  on  what  we  learned  through  our  interviews.  The  briefing 
closes  with  a  brief  discussion  of  implications  for  PBSA  elsewhere  in  the  Air  Force. 

Our  findings  remain  tentative  at  this  point.  As  we  present  this  material  to  other 
organizations,  we  are  looking  for  additional  insights.  We  are  especially  interested  in 
determining  whether  the  findings  reported  here  are  compatible  with  perceptions  held 
elsewhere  in  the  Air  Force.  We  are  also  interested  in  ascertaining  which  of  the  lessons 
learned  will  be  most  useful  to  report  to  a  broader  Air  Force  audience. 


PBS  A  Is  Endorsed  by  the 
Government  and  the  Air  Force 


*  Government  interest 

-  1991  OFPP  memo  stressed  use  of  PBSC 

-  FAR  changes  made  in  1997 

-  PBSA  goals  established  by  USD/A&T 

•  Air  Force  emphasis 

-  AFI 63-124  describes  AF  approach  to  PBSC 

-  ^'Lightning  Bolts”  address  some  PBSC  issues 

-  Some  command  publications  assist  implementation 
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In  April  1991,  the  Office  of  Federal  Procurement  Policy  (OFPP)  issued  Policy  Letter  91- 
2,  which  stated  that  it  was  the  policy  of  the  federal  government  that  agencies  use 
performance-based  contracting  methods  to  the  maximum  extent  practicable  when 
acquiring  services.  This  meant  “structuring  all  aspects  of  an  acquisition  around  the 
purpose  of  the  work  to  be  performed  as  opposed  to  either  the  manner  by  which  the  work 
is  to  be  performed  or  broad  and  imprecise  statements  of  work.”  The  Under  Secretary  of 
Defense  for  Acquisition  and  Technology  (USD/A&T)  reaffirmed  this  policy  in  April 
2000  by  directing  that  50  percent  of  service  acquisitions,  measured  both  in  dollars  and  in 
actions,  be  performance  based  by  the  year  2005. 

The  principles  outlined  in  OFPP  Policy  Letter  91-2  were  formally  incorporated  into  the 
FARs  in  1997.^  Other  changes  in  the  FARs  since  then  have  emphasized  the  importance 
of  using  market  research,  past-performance  information  as  an  element  of  proposal 
evaluation,  a  “best  value”  continuum  that  allows  for  the  consideration  of  factors  other 
than  price  in  a  source  selection,  and  practices  and  quality  assurance  plans  that  relate 
performance  to  incentives.® 

A  variety  of  Air  Force  instructions  (AFIs),  publications,  and  policies  direct  or  encourage 
the  use  of  performance-based  contracting  practices.  AFI  63-124,  Performance-Based 

''Each  service  was  directed  to  develop  a  plan  to  implement  this  policy.  The  Office  of  the  Deputy  Assistant 
Secretary  of  the  Air  Force  for  Contracting  (SAF/AQC)  released  the  Air  Force’s  Performance-Based 
Services  Acquisition  (PBSA)  Implementation  Plan  in  June  2000.  The  plan  can  be  accessed  at  the  Air  Force 
contracting  website:  http://www.safaq.hq.af.mil/contracting/toolldt/part37/. 

federal  Acquisition  Circular  (FAC)  97-01,  August  22, 1998.  FACs  are  most  easily  referenced  through  the 
Air  Force  FARSite  website  at  iittp://farsite.hill.af.mil/. 

^According  to  FAR  Part  2.101,  “‘Best  value’  means  the  expected  outcome  of  an  acquisition  that,  in  the 
Government’s  estimation,  provides  the  greatest  overall  benefit  in  response”  to  the  requirement.  For 
practices  related  to  PBSA,  see  FAR  Part  12,  “Acquisition  of  Commercial  Items”;  FAR  Part  15, 
“Contracting  by  Negotiation”;  FAR  Part  16,  “Types  of  Contracts”;  and  FAR  Part  37,  “Service 
Contracting.”  (FARs  are  also  most  easily  referenced  through  the  FARSite  website  given  above.) 
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Service  Contracts,  outlines  headquarters,  major  command  (MAJCOM),  and  installation 
responsibilities  in  structuring  acquisition  around  what,  as  opposed  to  how,  a  contractor 
should  perform  its  work.  Acquisition  policy  initiatives  called  “Lightning  Bolts,”  issued 
by  the  Assistant  Secretary  of  the  Air  Force  for  Acquisition  (SAF/AQ),  have  addressed 
some  PBSA  issues,  such  as  the  use  of  Acquisition  Support  Teams  (ASTs)  to  implement 
performance-based  concepts  during  acquisition  strategy  development.  Finally,  individual 
MAJCOM  documents,  such  as  Air  Mobility  Command’s  Performance-Price  Tradeoff 
Guide,  are  available  to  assist  contracting  offices  in  applying  new  or  unfamiliar  techniques 
to  the  acquisition  process.’ 


^See  Headquarters  Air  Mobility  Command,  Logistics  Directorate,  Contracting  (HQ  AMC/LGC), 
Performance-Price  Tradeoff  (PPT)  Gidde^  June  2000, 
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Study  Seeks  Information  on 
Successful  Implementations  to  Date 

•  Primary  goal 

-  Identify  examples  of  early  success  in  implementation 

-  Use  them  to  find  iessons  learned  on  howto  pursue 
PBSA  successfully  elsewhere— “build  on  success” 

-  Not  a  systematic  evaluation  of  current  practices 

•  Secondary  outcome:  “reality  check” 

-  What  is  being  done  differently  under  these  contracts? 

-  What  benefits  and  costs  can  be  measured? 

-  How  is  API  63-124  being  implemented? 

-  How  adequate  is  current  guidance  and  training? 
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The  primary  goal  of  this  study  is  to  use  examples  of  successful  Air  Force  applications  of 
PBSA  to  illustrate  how  to  pursue  these  practices  elsewhere.  It  is  decidedly  not  designed 
to  be,  nor  is  it,  a  systematic  evaluation  or  critique  of  PBSA  in  the  Air  Force.  A  secondary 
outcome  is  to  provide  a  “reality  check”  of  the  current  status  of  implementation  of  PBSA 
practices  to  help  determine  what  is  being  done  differently  under  PBSA,  how  the  costs  and 
benefits  of  new  practices  are  being  measured,  how  AFI 63-124  is  being  implemented,  and 
how  people  in  the  field  judge  the  adequacy  of  their  training  to  use  a  PBSA  approach. 
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Study  Focuses  on  Key  Aspects  of 
Innovative  Services  Acquisition 

•  Training 

•  Market  research 

•  Best  value  source  selection 

•  Outcome-oriented  SOWs 

•  Incentives  tied  to  outcome-oriented  performance 

•  Performance  management  plans 
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In  keeping  with  the  practices  described  in  policy  guidance  letters,  the  FARs,  and  Air 

Force  publications,  the  telephone  interviews  that  we  conducted  for  the  study  focused  on 

six  areas: 

•  The  availability  and  effectiveness  of  training  in  PBSA  practices.  Training  is  required 
before  anything  else  can  occur. 

•  Techniques  used  in,  and  the  effectiveness  of,  market  research.  Market  research 
identifies  commercial  practices  and  processes  that  help  the  Air  Force  understand  how 
to  define  and  monitor  performance,  choose  the  best  source  to  provide  such 
performance,  vmte  contracts,  and  provide  incentives. 

•  The  design  of  best  value  source  selection  policies  and  practices  to  select  sources  that 
the  Air  Force  trusts  to  perform  well  with  regard  to  its  statements  of  work  (SOWs), 
terms,  conditions,  and  performance  management  plans.  DoD  views  best  value  source 
selection  as  separate  from  PBSA. 

•  The  use  of  outcome-oriented  SOWs  that  include  measurable  performance  objectives 
tailored  to  meet  the  needs  of  the  customer  organization(s). 

•  Other  contract  terms  and  conditions  tailored  to  the  relevant  services  and  the  needs  of 
the  customer  organization(s),  including  incentives  that  are  tied  to  the  outcome-oriented 
performance  objectives  to  promote  improvement  over  time. 

•  The  development  of  performance  management  plans  that  are  tied  to  outcome-oriented 
performance  objectives,  including  effective  use  of  customer  satisfaction  ratings  in 
managing  provider  performance. 
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Study  Examines  22  Recent  Base 
Service  Contracts  on  15  Bases 
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We  talked  to  personnel  at  15  Air  Force  bases  in  five  commands — Air  Combat  Command 
(ACC),  Air  Education  and  Training  Command  (AETC),  Air  Force  Space  Command 
(AFSPC),  Air  Mobility  Command  (AMC),  and  U.S.  Air  Forces,  Europe  (USAFE) — ^and 
discussed  22  contracts  using  the  questionnaire  template  shown  in  Appendix  AETC 
and  AMC  were  heavily  represented  in  our  interviews.  As  shown  in  the  chart  above,  most 
of  the  contracts  are  related  to  MFH  maintenance,  grounds  maintenance,  custodial 
services,  food  services,  and  refuse/recycling.  The  ‘‘other”  category  includes  some  small 
contracts  for  special  services  as  well  as  a  large  contract  for  base  operating  services.  The 
base  operating  services  contract  was  a  planned  A-76  study  that  resulted  in  a  direct 
conversion.^ 

Except  for  the  USAFE  custodial  contract  and  the  “other”  contracts,  all  contracts  were 
either  small-business  or  8(a)  set-asides.^^  The  only  unrestricted  contracts  were  either  for 

^Appendix  C  lists  contracts  by  type  and  shows  the  positions  of  interview  participants.  The  contract 
administrator  and  an  appropriate  functional  representative  (e.g.,  someone  from  civil  engineering  or  the 
housing  ofBce)  were  usually  involved  in  the  discussion.  Participants  were  advised  that  all  answers  to 
questions  would  be  considered  confidential;  specific  information  in  this  report  associating  individual 
contracts  with  locations  has  been  included  only  with  the  permission  of  the  survey  participant. 

^A-76  refers  to  Office  of  Management  and  Budget  Circular  A-76,  which  prescribes  a  specific  (and  lengthy) 
cost  con:q)arison  process  to  determine  if  a  commercial  activity  currently  being  accomplished  by  employees 
of  the  government  should  be  outsourced  to  a  civilian  contractor.  Under  certain  circumstances,  a  “direct 
conversion”  can  be  accompHshed,  which  means  that  the  conversion  to  a  civilian  contractor  is  made  without 
going  through  the  A-76  process.  According  to  those  we  interviewed,  the  direct  conversion  was  justified  by 
arguing  that  the  base  had  insufficient  personnel  to  conduct  the  A-76  study  and  maintain  base  functions  at 
the  same  time.  Even  with  the  direct  conversion,  however,  the  process  involved  a  best  value  competition 
among  8(a)  contractors. 

^^Small-business  set-asides  are  procurements  that  are  reserved  exclusively  for  small  businesses  and  are 
authorized  in  certain  circumstances  by  the  Small  Business  Act.  One  of  several  restrictions  to  qualify  as  a 
small  business  in  the  services  field  is  that  annual  receipts  may  not  exceed  $2.5  to  $21.5  million,  depending 
on  the  service  being  provided.  So-called  8(a)  set-asides  (named  for  the  paragraph  in  the  Small  Business 
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technically  advanced  equipment  or,  in  the  USAFE  case,  for  services  outside  the  United 
States.  It  is  unclear  to  us  if  this  is  a  significant  fact  related  to  the  general  population  of 
suppliers  for  these  services  or  to  the  use  of  best  value  approaches. 

Two  contracts  were  awarded  in  FY  1998,  five  in  FY  1999,  and  fourteen  in  FY  2000 
(ei^t  of  those  were  in  November— December  1 999).  The  USAFE  contract  has  not  yet 
been  awarded. 

The  participants  in  our  interviews  were  self-selected.  The  Air  Force  Office  of 
Operational  Contracting  (SAF/AQCO)  sent  an  e-mail  to  all  contracting  offices  in  March 
2000  asking  for  examples  of  “success  stories”  of  performance-based  services  contracts, 
and  those  who  responded  did  so  on  the  basis  of  their  own  perception  that  they  were 
successful.”  From  the  group  of  respondents,  we  selected  those  that  covered  large 
contracts  (such  as  MFH)  and  that  allowed  us  to  examine  similar  services  in  different 
MAJCOMs,  as  well  as  several  additional  examples  that  promised  some  insight  into 
unusual  situations. 


Act  that  governs  them)  are  designed  for  small  businesses  owned  and  controlled  by  socially  or  economically 
disadvantaged  individuals. 

^^The  e-mail  asked  for  the  following  information:  dollar  savings  resulting  from  the  conversion  to 
performance-based  requirements,  improved  contractor  performance,  reduction  in  SOW  pages,  use  of 
commercial  standards  and  best  practices,  innovative  contractor  initiatives,  increased  customer  satisfaction, 
reduced  need  for  contract  modifications,  and  reduction  in  the  amount  of  surveillance  required.  It  also 
invited  comments  about  problems  encountered  with  performance-based  contracts. 
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INTERVIEW  RESULTS 

Let  us  now  turn  to  the  results  themselves. 
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Walking  Through  the  Process 

•  Motivation  for  implementing  PBS  A 

•  Getting  started 

-  BRAG 

-  Training 

-  Market  research 

-  Defining  requirements 

•  The  contract 

-  SOW  development 

-  Quality  Assurance  Surveillance  Plan  (QASP) 
development 

-  Contract  structure 

-  Source  selection  process 

-  Past  performance  data 

RAND  PnftetMItmKE _  S 


Our  interviews  were  structured  around  the  process  for  implementing  performance-based 
service  contracts,  and  the  discussion  that  follows  divides  this  process  into  two  phases. 
The  “getting  started”  phase  includes  the  establishment  of  a  multifunctional  team  (called  a 
Business  Requirements  Advisory  Group  [BRAG]  in  API  63-124)  to  guide  the  whole 
process,  training  personnel  in  PBSA  techniques,  organizing  market  research,  and 
finalizing  the  requirements  for  the  desired  service.  The  “contract  phase”  in  our 
discussion  starts  with  development  of  the  SOW  and  the  Quality  Assurance  Surveillance 
Plan  (QASP)  to  describe  the  nature  of  the  work  required  and  how  performance  will  be 
assessed.  The  type  of  contract  to  be  used  and  the  nature  of  the  competition  for  the 
contract  are  determined  before  the  solicitation,  and  the  use  of  past-performance  data  is 
important  in  the  award  of  the  contract.  Many  of  the  individual  steps  of  the  process  listed 
in  this  chart  overlap;  the  order  of  discussion  is  for  convenience  only  and  is  not  meant  to 
imply  a  chronological  sequence. 
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Motivation  for  impiementing  PBS  A 
Varied 

•  Headquarters  had  an  important  role 

-  ACC  and  USAFE  bases  were  directed  to  use  PBSA 

-  Three  from  AMC,  two  from  AETC  noted  HQ  directives 

•  Prior  experience  affected  attitudes 

-  Three  from  AMC  and  two  from  AETC  were  dissatisfied 
with  previous  contractors 

-  One  from  AETC  thought  'timing  was  righr  for  PBSA 

•  Individual  initiative  was  also  important 


RAND  PitjtciAIRFORCe _ ^ 


The  majority  of  those  we  talked  to  said  that  they  used  performance-based  practices 
because  they  had  been  directed  to  do  so  by  higher  headquarters  or  because  of  the  release 
of  API  63-124.  However,  several  people  in  the  commands  that  were  “pushing”  PBSA 
emphasized  that  their  dissatisfaction  with  the  performance  of  previous  contractors  had  led 
them  to  consider  using  a  different  approach  for  awarding  contracts.  They  hinted  that  they 
would  have  tried  something  new  even  without  headquarters  direction  because  of  their 
unhappiness  with  lowest  bidders. 

Work  on  the  “oldest”  contract  we  discussed  in  om  interviews  was  started  in  early  1998. 
Because  the  contract  needed  to  be  awarded  in  a  very  short  time  in  order  to  meet  a  critical 
need,  the  contract  manager  was  encouraged  to  “push  the  boundaries”  of  the  FARs  to 
complete  the  process  quickly.  The  manager  responded  creatively  and  applied 
performance-based  practices  effectively  without  expUcit  MAJCOM  guidance. 
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Multifunctional  Teams  Helped  People 
Get  Started 

•  Multifunctional  teams  were  used  by  most  bases 

-  Membership  on  teams  varied 

~  Frequency  of  meetings  varied  widely 

-  “Formal”  BRAGs  were  not  the  only  approach 

•  Teams  were  effective 

-  Teamwork  increased  efficiency 

-  Teams  contributed  to  sense  of  collective  “ownership” 


RAND 


A  Business  Requirements  Advisory  Group  must  be  established  for  every  requirement  that 
operates  under  API  63-124.  A  BRAG  is  a  “customer-focused,  multifunctional  team”  that 
manages  a  service  contract  throughout  the  life  of  the  requirement  and  supports  best  value 
business  decisions,  acquisition  planning,  market  research,  source  selection,  and  contract 
performance  management.  Those  involved  with  10  of  our  22  contracts  said  that  they 
used  a  “formal”  BRAG.  Five  said  they  had  equivalent  groups  (most  for  contracts 
developed  before  AFl  63-124  was  released);  one  said  that  its  Acquisition  Strategy  Panel 
(ASP)  served  as  the  BRAG.^^ 

People  we  interviewed  did  not  always  provide  specific  information  on  the  membership  of 
the  group;  in  general,  they  noted  that  the  contracting  office  and  the  primary  functional 
area  were  always  represented.  Legal,  financial,  and  quality  assurance  evaluation  (QAE) 
personnel  were  also  represented.  Participation  by  higher-ranking  personnel  (e.g.,  the 
commander  of  the  contracting  squadron)  was  rare;  as  one  person  said,  “It’s  hard  to  get 
face  time  with  the  squadron  commander.”  Although  AFI  63-124  says  that  installation 
commanders  are  to  provide  the  resources  necessary  for  BRAGs  to  carry  out  their  duties, 
personnel  at  one  base  noted  their  concern  that  BRAGs  could  present  a  problem  if  the  base 
is  low  on  manpower,  since  people  have  to  do  their  ovm  jobs  in  addition  to  working  on  a 
BRAG. 

Meeting  frequency  varied  widely.  The  general  attitude  was  that  teams  met  “as  needed.” 
At  one  base,  people  said  they  expected  the  “full  BRAG”  to  meet  only  annually  to  satisfy 
their  interpretation  of  the  AFI.  At  another,  the  BRAG  met  eight  hours  a  day  for  a  week  to 
develop  a  SOW. 

Although  there  was  occasional  cynicism  about  the  BRAG  (we  heard  remarks  such  as 
“It’s  just  a  name  change,”  “We  filled  the  square,”  and  “It  ended  up  looking  like  meetings 


^^ASPs  are  used  to  “develop  a  systematic  and  disciplined  approach  to  achieve  an  efficient/effective 
acquisition.”  Air  Force  FAR  supplement  (AFFAR)  5307.104-91  describes  when  they  are  used  and  how  to 
conduct  them. 
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under  the  previous  style  of  contracting”),  there  was  genuine  enthusiasm  for  the 
effectiveness  of  a  cross-functional  team,  whether  it  is  called  a  BRAG  or  something  else. 
At  one  AMC  base,  the  BRAG  cut  the  time  required  for  many  activities  in  half,  because 
information  related  to  the  contract  was  shared  in  real  time  rather  than  in  written 
memoranda,  and  at  an  AETC  base  the  BRAG  limited  the  time  wasted  going  back  and 
forth  between  different  offices.  Personnel  at  a  third  base  noted  that  the  BRAG  helped  the 
contracting  office  better  understand  the  functional  customer’s  needs — in  fact,  it  helped 
them  in  their  market  research  for  the  contract. 

Effective  teamwork  and  the  feeling  of  “ownership”  of  the  process  were  noted  several 
times.  The  BRAG  helps  the  contracting  office  and  the  functional  customer  feel  that 
eveiyone  is  working  together  to  ensure  that  problems  are  being  solved.  Problems 
associated  with  acquiring  services  are  no  longer  just  “contracting’s  job”  or  “contracting’s 
fault.” 
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Ineffective  Training  Made 
Impiementation  More  Difficult 

•  Limited  availability  was  an  issue 

-  Many  bases  had  no  training 

-  Training  was  often  limited  to  single-day  awareness 
courses 

•  Available  training  was  often  ineffective 


RAND 


Those  contracts  we  examined  were  developed  when  the  push  for  PBSA  was  new,  so 
difficulties  with  training  for  the  approach  might  be  expected.  However,  general 
frustration  was  expressed  with  the  lack  of  training  provided  before  contracts  using  PBSA 
were  required. 

Personnel  at  five  bases  representing  AMC  and  AETC  said  they  had  no  training  at  all — 
and  one  of  the  contracts  involved  was  awarded  as  late  as  January  2000.  Those  who  said 
they  had  at  least  some  training  generally  referred  to  assistance  from  headquarters  or  some 
on-site  awareness  training.  Headquarters  ACC  conducted  a  performance-price  tradeoff 
(PPT)  workshop  at  the  ACC  base,  and  a  private  firm  was  contracted  to  teach  a  PPT 
course  for  personnel  at  one  of  the  AMC  bases.  It  is  encouraging  to  note  that  personnel  at 
three  bases  (two  from  AMC  and  one  from  USAFE)  that  have  awarded  contracts  since 
January  2000  reported  having  received  more  useful  training,  so  the  situation  is  better  than 
it  was.  However,  even  these  people  saw  room  for  improvement  in  training  programs. 

The  biggest  complaints  were  that  training  was  too  short  (one  day)  or  was  ineffective 
because  it  provided  an  overview  of  PBSA  practices  instead  of  a  detailed  discussion  of 
how  to  apply  those  practices.  Videotapes  were  considered  boring;  a  CD-ROM  produced 
by  the  Air  Force  Civil  Engineer  Support  Agency  (AFCESA),  however,  was  judged  to  be 
very  good  by  the  AMC  base  that  received  it. 

We  will  return  to  this  topic  at  the  close  of  the  briefing  to  suggest  changes  that  could 
improve  Air  Force  training  on  PBSA. 
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Approaches  to  Market  Research 
Varied 

•  Different  methods  were  used 

-  Media  range  from  Yellow  Pages  to  the  Internet 

-  Telephone  and  mall  surveys  had  limited  success 

-  Site  visits  to  commercial  firms  were  useful 

•  Who  should  do  it  is  a  matter  of  discussion 

-  Contracting  personnel  seemed  to  do  most  of  it 

~  Cooperation  of  functionals,  “customers”  necessary 

•  The  time  to  accomplish  it  varies 

•  It’s  a  continuous  process 

RAND  ht^aefAIKFOPCB _ ^ 


A  wide  variety  of  approaches  was  used  in  market  research,  ranging  from  thumbing 
through  the  Yellow  Pages  to  conducting  structured  interviews  to  engaging  in  site  visits  to 
commercial  firms  with  similar  service  needs. 

Personnel  at  several  bases  mentioned  that  they  had  had  little  success  gathering 
information  by  sending  questionnaires  to  commercial  buyers  or  providers.  Large 
providers  in  particular  were  not  interested  in  providing  assistance  unless  they  had 
previous  experience  with  Air  Force  contracts.  In  one  case,  a  provider  expected  to  be  paid 
for  answering  questions  in  the  market  research  phase. 

Those  who  had  visited  commercial  firms  that  bought  the  services  in  question  seemed  to 
be  most  satisfied  with  the  cooperation  they  received,  and  for  the  most  part  searches  of  the 
web  were  successful  and  productive.  In  at  least  two  cases  (one  MFH  and  one  custodial), 
the  contracting  office  had  found  it  difficult  to  locate  commercial  buyers  whose 
experience  included  services  of  the  size  and  scope  needed  by  the  base. 

Experience  with  government  and  Air  Force  websites  as  sources  of  information  or  advice 
for  market  research  varied,  and  there  was  not  an  obvious  “maturing”  of  the  users  or  the 
websites  over  time.  For  example,  those  who  developed  an  AETC  contract  awarded  in 
early  1998  found  a  web  search  valuable,  but  another  AETC  group  working  in  mid- 1999 
found  most  web  information  out  of  date  (although  they  did  find  llie  Air  Force  Materiel 
Command  [AFMC]  website  useful).  People  at  one  AMC  base  had  a  positive  experience 
with  the  web  in  developing  a  food  services  contract  in  mid- 1999,  but  a  group  developing 
an  MFH  contract  at  another  AMC  base  a  few  months  later  found  even  the  AMC  website 
of  limited  usefulness. 

Despite  occasional  dissatisfaction  with  electronic  media,  there  was  general  agreement 
that  informal  Air  Force  networks  were  helpful  for  exchanging  information. 

In  most  of  our  discussions,  it  was  clear  that  the  contracting  squadron  was  the  office 
primarily  responsible  for  conducting  market  research.  At  only  two  bases  did  those  we 
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interviewed  specifically  note  that  this  was  a  problem  or  indicate  some  resentment,  but  a 
comment  from  one  base  is  worth  quoting  in  full: 

Make  sure  that  the  functional  customer  and  contracting  work  together  in 
performing  the  market  research.  Market  research  is  the  key  to  the  whole 
thing — without  it,  there  is  no  basis  for  the  SOW...CE  must  be  drawn  in, 
even  if  it  is  reluctant.  Contracting  and  the  functional  have  very  different 
questions  to  ask  in  market  research.  Both  need  to  participate  to  make  a 
questionnaire  complete. 

The  time  required  to  accomplish  market  research  (for  those  who  were  specific  about 
times)  varied  widely:  One  grounds  maintenance  group  took  three  weeks  to  do  it;  another 
took  four  to  five  months. 

There  were  three  highly  encouraging  things  to  note  about  market  research: 

•  First,  even  those  who  complained  about  the  lack  of  guidance  available  in  this  area  had 
little  difficulty  accomplishing  the  research  successfully.  Everyone  took  the' initiative 
to  explore  a  variety  of  sources  to  get  information  that  they  found  useful  for  developing 
the  contract. 

•  Second,  those  who  took  a  lot  of  time  to  conduct  their  research  attributed  their 
“slowness”  to  lack  of  experience  and  felt  that  the  process  would  be  much  simpler  with 
the  next  contract  they  worked  with. 

•  Finally,  personnel  at  several  bases  mentioned  that  market  research  was  a  continuing 
process  (one  said  “never  ending”) — especially  continuing  through  contract 
development.  The  contracting  office  at  one  base  compiles  a  market  research 
document  that  users  consider  to  be  a  “living”  document. 
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Guidance  on  Defining  Requirements 
Was  Limited 


•  Few  bases  mentioned  guidance  in  the  form  of 
MAJCOM  instructions 

-  Earlier  work  led  us  to  believe  that  PBSA  might  require 
waivers  or  adjustments  to  other  APIs 

•  MAJCOM  guidance  in  one  case  seems  to  go 
against  the  “spirit”  of  PBSA  implementation 


RAND  MeetMRFORCe 


Our  earlier  research,  as  well  as  our  interactions  with  officers  involved  with  contracting, 
had  led  us  to  believe  that  full  implementation  of  PBSA  might  require  broad  changes  to 
APIs  and  other  formal  Air  Force  documents.  For  example,  in  a  briefing  presented  at  the 
annual  CE/Contracting/Industry  Partnering  workshop  in  February  2000,  an  AFMC 
representative  said  that  AFI 63-124  wasn’t  as  easy  to  use  as  many  had  thought  it  would 
be}^  This  individual  suggested  that  when  a  base  tries  to  apply  commercial  standards  to  a 
contract,  the  Air  Force  needs  to  provide  more  guidance  on  which  APIs  can  be  waived  and 
which  ones  are  mandatory.  During  discussion  of  this  issue,  some  participants  said  that 
AFIs  were  themselves  flexible  enough  to  allow  PBSA  practices  without  requiring 
waivers.  Others  believed  that  many  AFIs  needed  to  be  rewritten  to  reflect  commercial 
standards,  and  one  participant  recommended  that  all  AFIs  related  to  service  standards 
should  be  eliminated.  Similar  views  were  evident  in  early  discussions  of  the  Pick-a-Base 
program  in  AETC,  which,  among  other  things,  takes  an  aggressive  approach  toward 
PBSA.^"^ 

Although  these  discussions  have  been  in  the  context  of  A-76  studies,  it  is  clear  that  there 
has  been  disagreement  about  the  flexibility  that  is  allowed  when  performance-based 
requirements  are  defined.  In  light  of  this  disagreement,  it  was  striking  to  us  that  with  the 
exception  of  one  MAJCOM,  no  mention  was  made  of  formal,  written  command  guidance 

O.  Butler,  AFMC/CEIO  (Civil  Engmeering,  Organization,  and  Privatization),  “AFMC  Lessons 
Learned,”  briefing  presented  at  the  annual  CE/Contracting  /Industry  Partnering  workshop  at  Tyndall  Air 
Force  Base,  Florida,  on  February  23-24, 2000. 

^'‘The  AETC  Pick-a-Base  strategy  is  an  approach  toward  incorporating  multiple  support  fimctions  at  one 
base  into  one  A-76  study.  It  is  designed  to  “maximize  cross-toctional  synergy”  and  to  encourage 
innovation.  According  to  an  AETC  information  paper  by  Lieutenant  Colonel  Randy  Coleman,  “Basically, 
Pick-a-Base  endeavors  to  consolidate  base  operating  support  into  as  few  contracts  or  Most  Efficient 
Organizations  as  feasible.”  See  Randy  Coleman,  “Pick-a-Base:  An  Alternative  Approach  to  CS&P,”  Air 
Education  and  Training  Command,  April  30, 1999,  available  at  http://public.afca. scott. 
af.mil/pubiic/99  May/may08.htm. 
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related  to  PBSA  implementation,  and  there  were  no  requests  for  API  waivers  for  any  of 
the  22  contracts  we  discussed. 

The  contracting  office  at  one  AMC  base  complained  specifically  about  a  lack  of  formal 
guidance  for  a  contract  solicited  in  mid- 1999  and  awarded  in  early  2000:  “We  developed 
our  PBSCs  to  [meet]  the  new  Air  Force  Civil  Engineer  Support  Agency  standards.  When 
we  sent  them  to  AMC  headquarters  for  review,  they  applied  different  standards,  which 
they  had  not  written  down  and  shared  at  the  time.” 

AETC  was  the  exception  to  the  lack  of  formal  guidance,  with  published  (and  fairly  strict) 
standards  for  what  level  of  service  is  allowable  in  grounds  maintenance,  custodial,  and 
refuse/recycling  contracts.  AETC  headquarters  tells  bases  that  they  can  buy  only  the 
minimum  level  of  service  required  to  meet  government  goals,  and  that  it  will  review  large 
contracts  to  verify  compliance.  A  base  that  seeks  to  establish  higher  standards  must  pay 
for  them  by  diverting  discretionary  funds  that  could  be  used  elsewhere.  One  base  that 
receives  frequent  visits  by  distinguished  visitors  pays  for  extra  grounds  maintenance 
work  during  those  visits — and  during  special  events — with  its  own  operations  and 
maintenance  (O&M)  money.  Another  base  wanted  roadside  trash  collection  done  several 
times  a  week,  but  the  AETC  standard  called  for  collection  once  every  two  weeks.  The 
base  was  unwilling  (or  unable)  to  use  discretionary  funds  for  more  frequent  service. 
Another  base  did  not  combine  refuse  and  recycling  services  because  AETC  standards 
would  not  allow  it. 

We  asked  personnel  at  one  base  if  it  was  possible  to  reconcile  AETC’s  establishment  of 
minimum  requirements  for  which  it  would  pay  with  contract  offers  that  went  beyond  the 
minimum.  We  were  assured  that  this  was  not  a  problem  and  were  informed  that  a 
contractor  could  still  get  a  “blue”  rating  in  this  case.*^  However,  the  potential  for  conflict 
between  PBSA  implementation  and  strict  “minimal”  standards  seems  clear.  A  contractor 
that  could  take  advantage  of  economies  of  scope  by  combining  refuse  and  recycling  is 
apparently  unable  to  offer  to  do  so  at  an  AETC  base;  both  the  Air  Force  and  the 
contractor  lose  in  this  case.  It  is  also  easy  to  imagine  a  scenario  in  which  one  contractor 
offers  to  perform  well  above  the  AETC  standard  at  a  slightly  higher  price  than  a 
contractor  that  just  meets  the  minimum  standard.  The  Air  Force  might  not  get  “best 
value”  if  a  base  is  faced  with  the  choice  of  either  paying  out  of  its  own  pocket  to  get  the 
higher  performance  or  settling  on  the  poorer  performer  that  just  meets  the  AETC 
standard. 


'^In  Air  Force  evaluations,  technical  aspects  of  a  proposal  are  granted  a  color  rating  of  blue,  green,  yellow, 
or  red.  A  blue  rating  means  that  the  proposal  “exceeds  minimum  performance  or  capability  requirements  in 
a  way  beneficial  to  the  Air  Force.”  This  is  described  in  AFFAR  53-15,  paragraph  5315.305. 
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SOW  Development  Was  More  Difficult 
but  Was  Worth  the  Effort 

•  New  SOWS  required  more  work 

-  New  approaches  to  writing  them  are  needed 

•  Reduction  in  SOW  length  can  be  substantial 

-  Some  had  huge  reductions^hundreds  of  pages 

•  Several  factors  led  to  reductions 

-  sow  language  emphasized  ^^A^iat”  instead  of  “how 
to” 

-  Formerly  required  “boilerplate”  was  reduced 

-  References  to  commercial  practices  increased 


RAND  Pn^MftPORCe _ « 


Most  of  those  we  interviewed  said  that  performance-based  SOWs  took  more  time  to 
develop  because  the  writers  were  inexperienced  and  there  were  few  examples  available  to 
work  from.  People  at  one  base  said,  “We  used  to  have  a  boilerplate  SOW,  give  it  to  the 
functional  customer,  and  go  back  and  forth  a  couple  of  times.  Now. .  .we  work  together 
to  do  the  market  research,  which  dictates  the  contents  of  the  SOW.”  Without  the 
boilerplate,  writing  from  scratch  was  sometimes  difficult — especially  because  functional 
customers  (particularly  QAEs)  were  forced  to  “rethink”  their  approach  to  writing  the 
SOW,  and  there  was  discomfort  with  the  lack  of  direction  to  the  contractor  on  how  to  do 
things.  However,  this  experience  was  not  universal;  personnel  at  some  bases  thought  that 
writing  their  performance-based  SOWs  was  less  difficult  because  the  task  was  done  in 
the  context  of  a  BRAG,  where  communication  and  cooperation  were  easier. 

Although  most  interviewees  said  their  job  was  made  more  difficult  by  the  lack  of  sample 
performance-based  SOWs,  some  said  they  were  able  to  build  on  the  work  of  other  bases 
in  their  command.  Others  noted  that  they  received  assistance  from  command 
headquarters,  but  this  did  not  always  make  the  job  easier  because  shifting  command 
perspectives  resulted  in  an  iterative  process  to  create  an  acceptable  document. 

The  results  of  writing  performance-based  SOWs  were  in  some  cases  remarkable.  One 
MFH  SOW  was  reduced  from  over  300  pages  to  just  under  40,  and  several  other  bases 
cut  more  than  100  pages  from  their  SOWs.  People  at  virtually  every  base  said  that  their 
SOWs  now  emphasize  the  outcome  desired  without  going  into  details  of  how  to  do  it, 
adding  that  their  SOWs  are  also  shorter  because  they  eliminate  needless  definitions  by 
referencing  commercial  standards.  A  review  of  several  of  these  contracts  confirmed  that 
these  assessments  are  accurate;  reductions  are  not  simply  a  result  of  API  63-124  allowing 
references  to  other  APIs  instead  of  “boilerplate”  language. 

The  Professional  Groimds  Maintenance  Society  (PGMS)  and  the  American  Society  of 
Landscape  Architects  provide  common  groimds  maintenance  standards,  and  several  bases 
referenced  the  Building  Owners  and  Managers  Association  (BOMA)  for  custodial 


22 


services  standards.  One  grounds  maintenance  example  of  the  “old”  way  of  stating 
requirements  in  the  SOW  is  “Use  a  mulching  mower  set  to  no  more  than  four  inches  on 
Monday  and  Thursday;  trim  on  the  same  day  as  mowing.”  The  current  SOW  says  only 
“Mow  improved  areas  using  PGMS  standards.” 

Market  research  revealed  that  in  certain  places,  commercial  firms  tended  to  use  standards 
more  attuned  to  the  locale.  For  example,  one  grounds  maintenance  contract  reflected 
input  from  a  local  agricultural  extension  service  office. 
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Customer  Complaints  Were  More 
Important  in  QASPs 

•  Few  explicit  applications  of  commercial  standards 

•  Focus  on  customer  complaints 

-  Transitioning  from  inspections  to  full  reliance  on 
complaints  will  take  some  time 

•  Intentional  shift  of  responsibilities  to  the  vendor 

•  PBSC  approach  adds  sense  of  partnership, 
improves  communication 

RAND  _ 16_ 


In  our  interviews,  few  explicit  comments  were  made  about  applying  commercial 
standards/approaches  to  QASPs,  although  there  were  occasional  remarks  to  the  effect  that 
customer  complaints  were  the  primary  method  of  quality  control  in  the  commercial 
world.  Persoimel  at  one  base  said  that  in  their  research  they  had  asked  specific  questions 
about  how  the  commercial  world  does  everything — payments,  inspections,  evaluation 
factors — and  in  the  process  had  discovered  a  commercial  software  tool  that  will  be  useful 
in  their  quality  control  plan. 

All  of  the  contracts  we  discussed  in  our  interviews  have  shifted  the  focus  of  their  quality 
assurance  programs  fi'om  regular  inspections  by  QAEs  to  reliance  on  customer 
complaints*^ — although  the  new  focus  is  not  always  that  sharp.  A  few  seemed  somewhat 
inconsistent  in  oin  discussions;  they  would  say  that  they  rely  on  customer  complaints  but 
would  then  talk  about  using  100  percent  inspections,*^  so  it  was  not  clear  what  they  were 
actually  using.  Invalid  customer  complaints  were  an  issue  as  well,  and  all  bases 
mentioned  the  importance  of  customer  education  to  ensure  understanding  of  vendor 
obligations  xmder  the  contract.  We  heard  repeatedly  of  transitional  periods  early  in 
contracts  during  which  Air  Force  personnel  complained  about  things  not  covered  in  the 
contract,  but  after  customer/ftmctional  training  was  accomplished,  the  ratio  of  valid 
complaints  to  the  total  munber  of  complaints  rose  markedly.  It  appears  that  proactive 
training  eliminated  this  transitional  experience,  and  the  importance  of  such  training  was 
recognized.  For  example,  a  custodial  QASP  we  received  outlined  training  for  building 
managers  that  included  discussions  about  the  requirements  of  the  contract,  how  to  make  a 


Again  commercial  firms  and  the  Air  Force  tend  to  think  about  “customers”  differently.  In  the  Air  Force, 
the  “customers”  empowered  to  complain  are  sometimes  the  ultimate  customers — ^i.e.,  the  actual  users  of  the 
services  bought — and  are  sometimes  the  Air  Force  functionals  responsible  for  day-to-day  oversight. 
Commercial  firms  tend  to  focus  much  more  on  customers  who  actually  consume  a  service  bought  firom  an 
external  source. 

'^This  means  that  each  time  work  is  performed,  it  is  evaluated  or  inspected. 
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complaint,  and  whom  to  contact  in  the  government  in  the  event  that  a  complaint  was  not 
resolved. 

AFI  63-124  allows  methods  of  surveillance  to  be  adjusted  after  contract  award,  and  the 
ACC  base  has  explicitly  implemented  this  approach:  Although  its  QAEs  now  do  100 
percent  inspections,  the  QASP  states  that  this  approach  can  be  modified  on  the  basis  of 
contractor  performance  with  the  hope  of  eventually  relying  on  customer  complaints 
alone.  Evaluators  at  other  bases  noted  that  the  dependence  on  inspections,  as  well  as  the 
frequency  of  inspections,  varied  with  vendor  experience  and  with  the  nature  of  the 
contract  (e.g.,  some  environmental  and  health-related  areas  might  always  require  100 
percent  inspection). 

Personnel  at  five  bases  specifically  mentioned  shifting  the  burden  of  compliance/quality 
control  to  the  contractor  in  order  to  reduce  surveillance  costs,  and  we  got  the  sense  from 
the  other  bases  that  this  was  their  attitude  as  well.  One  AMC  base  has  a  custodial 
contractor  that  is  ISO  9000  compliant  (certified  as  ISO  9002).^^  This  certification 
requires  that  the  vendor  undergo  unannounced  audits  as  well  as  semiannual  inspections. 
This  ‘‘external”  third-party  quality  control  reduces  the  monitoring  burden  for  the  base. 

An  important,  and  frequent,  comment  from  the  field  was  that  the  shift  from  “oversight”  to 
“insight”  through  customer  complaints  contributes  to  a  partnering,  rather  than  an 
adversarial,  relationship.^^  We  heard  remarks  such  as  “Now  CE  can  talk  directly  to  the 
contractor... [to]  get  a  solution”;  “The  contractor  wants  to  do  the  right  thing”;  and  “In  the 
past,  contracting  might  communicate  by  formal  letter;  now  they  will  use  informal 
telephone  calls.” 


*^The  International  Organization  for  Standardization  (ISO)  is  a  worldwide  federation  of  national  standards 
bodies  from  some  130  countries.  According  to  the  organization,  ‘The  ISO  9000  family  of  standards 
represents  an  international  consensus  on  good  management  practices  with  the  aim  of  ensuring  that  the 
organization  can  time  and  time  again  deliver  the  product  or  services  that  meet  the  client’s  quality 
requirements.  These  good  practices  have  been  distilled  into  a  set  of  standardized  requirements  for  a  quality 
management  system,  regardless  of  what  your  organization  does,  its  size,  or  whether  it’s  in  the  private,  or 
public  sector.”  See  the  ISO  website,  http://www.iso.ch,  for  details. 

’^In  a  PBSA  training  briefing  by  the  Air  Force  Logistics  Management  Agency  Contracting  Division 
(AFLMA/LGC),  the  difference  is  described  in  this  manner:  “Oversight  usually  meant  [that]  the  customer, 
or  QAE,  aimed  inspections  at  detecting  if  there’s  a  problem;  in  other  words  we  kept  an  eye  on  the 
contractor.  Now,  we’re  asking  our  QA  personnel  to  review  contractor  metrics.  That  is,  review  the  metrics 
with  an  understanding  and  validating  role  to  monitor  the  contractor’s  quality  surveillance  system.  The  QA 
person  is  not  reviewing  for  performance;  he  or  she  is  reviewing  for  good  management.  We  are  no  longer 
targeting  the  process  as  the  primary  focus:  we  are  targeting  results.” 
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Most  Contracts  Were  FFP  and  of 
“Standard”  Duration 

•  Contract  types  limited 

-  All  but  three  are  firm  fixed  price  (FFP) 

•  Terms 

-  Contracts  with  a  basic  period  plus  four  option  years 
predominate 


RAND  Pn^MRfORCe _ _ _ !L 


All  but  three  of  the  contracts  in  this  study  are  firm  fixed  price  (FFP).  The  refuse  contract 
is  fixed  price  plus  economic  price  adjustment  because  of  anticipated  increases  in  the 
landfill  fee  (the  landfill  is  operated  by  the  local  municipality).  Because  of  the  large  work 
scope  and  expected  changes  in  base  activities,  the  AFSPC  base  decided  on  a  cost 

contract.  One  base  used  an  indefinite-delivery,  indefinite-quantity  (IDIQ)  contract  for  a 
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varying  staffing  requirement. 

Contract  terms  with  a  basic  period  plus  four  option  years  predominated,^^  with  two 
interesting  exceptions.  The  refuse  contract  required  the  purchase  of  new  equipment,  and 
the  base  extended  the  contract  period  for  two  years  in  order  to  allow  the  vendor  to 
amortize  the  capital  improvement  costs  over  a  longer  period. 

One  AETC  custodial  contract  was  slightly  unusual  as  well.  The  previous  contractor  was 
terminated  for  poor  performance  after  a  year  and  a  half.  The  current  contract  uses  five 
six-month  options  to  allow  for  integration  into  a  forthcoming  A-76  study. 


FFP  contract  provides  a  price  that  is  not  subject  to  adjustment  on  the  basis  of  the  contractor’s  cost 
experience  in  performing  the  contract.  A  fixed-price  contract  with  economic  adjustment  provides  for 
revisions  of  the  stated  contract  price  in  the  event  of  specified  contingencies.  A  cost  contract  provides  for 
the  payment  of  allowable  incurred  costs  as  prescribed  in  the  contract.  IDIQ  contracts  allow  for  flexibility 
in  the  quantity  and  timing  of  services  delivered.  FAR  Part  12.207  says  that  agencies  shall  use  FFP  or  fixed 
price  with  economic  adjustment  in  contracts  for  the  acquisition  of  “commercial”  items.  IDIQ  may  be  used 
^^dlere  the  prices  are  established  on  the  basis  of  an  FFP  or  fixed  price  with  economic  price  adjustment, 
adding  that  “use  of  any  other  contract  type  to  acquire  commercial  items  is  prohibited.”  FAR  Part  37.602-4 
says,  “Contract  types  most  likely  to  motivate  contractors  to  perform  at  optimal  levels  shall  be  chosen.” 

Service  contracts  are  structured  with  a  “basic  period”  and  “option  years”  because  continuation  of  a 
contract  is  subject  to  appropriations  fi-om  Congress.  FAR  Part  17.204(e)  discusses  the  usual  standard  of 
one  basic  year  plus  four  option  years.  It  is  generally  understood  that  the  contract  will  continue  in  the  option 
years  unless  the  contractor  performs  unsatisfactorily.  Later  we  will  address  award  terms,  which  reward  a 
contractor  for  excellent  performance  by  extending  the  originally  negotiated  length  of  the  contract. 
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Limited  Use  of  Award  Fees/T erms^ 
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“This  chart  lists  all  22  contracts  to  show  how  many  had  award  fees, 
award  terms,  or  neither.  “1F&T’‘  means  there  was  one  contract  with 
both  an  award  fee  (F)  and  an  award  term  (T).  “2F’’  means  two 
contracts  had  award  fees.  A  number  by  itself  means  that  there  was 
no  award. 

There  were  varying  opinions  on  award  fees  and  terms.  Most  contracting  officers 
involved  with  custodial  contracts,  for  example,  said,  “How  can  you  be  cleaner  than 
clean?”  and  therefore  appeared  to  question  how  an  award  could  be  equitably 
administered. 

Personnel  at  some  bases  mentioned  that  they  had  discussed  the  possibility  of  award  fees 
but  decided  that  funds  would  not  be  available  for  them.  Those  at  AMC  bases  felt  that 
grounds  maintenance  was  a  “high  visibility”  item  that  warranted  the  incentive  of  award 
fees  and  were  successful  in  arguing  for  them.  Officials  at  the  ACC  base  had  to  fight  with 
headquarters  to  get  the  award  fee  for  MFH,  because  headquarters  thought  the  use  of  past- 
performance  information  in  awarding  future  contracts  should  be  motivation  enough. 
Eventually,  the  “visibility”  argument — in  addition  to  the  importance  of  MFH  for  quality 
of  life — convinced  ACC  headquarters  to  allow  the  award  fee. 

The  award-fee  criteria  were  usually  a  combination  of  objective  measures  (such  as 
statistical  comparisons  of  performance  with  requirements  in  the  SOW)  and  subjective 
measures  (a  committee’s  assessment  of  “quality”),  and  in  all  cases  the  criteria  were 
clearly  outlined  for  the  vendor. 

Our  interviews  revealed  that  most  bases  had  not  considered  using  award  terms.  Some  of 
those  interviewed  said  they  could  not  find  examples  of  commercial  use  of  such  terms;^" 

fact,  formal  award  terms  are  not  common  in  commercial  contracting.  However,  an  ad  hoc 
administrative  decision  by  a  commercial  buyer  to  extend  a  contract  with  a  good  provider  is  a  common 
phenomenon.  The  rationale  is  this:  Why  change  horses  in  midstream  if  your  current  mount  is  a  good  one? 
An  award  term  can  be  seen  as  a  government  proxy,  tailored  to  the  more  formal  administrative  structure  that 
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others  felt  that  knowing  that  performance  would  be  evaluated  in  future  competitions  was 
sufficient  incentive  for  vendors  to  perform  well.  The  motivation  for  the  ACC  base’s  use 
of  award  terms  is  unclear,  but  administration  of  the  award  is  reasonably  well  laid  out  in 
an  award-term  plan.  After  the  second  year  of  the  contract,  a  review  of  contractor 
performance  with  respect  to  award  fees  will  be  conducted;  that  information  combined 
with  subjective  assessments  of  other  performance  areas  will  determine  if  the  contract  will 
be  extended  to  a  sixth  year  (one  year  beyond  the  standard  basic  year  plus  four  option 
years).  After  the  third  year  of  the  contract,  another  review  will  be  conducted  to 
determine  the  award  of  a  seventh  year. 


federal  contracting  requires,  for  this  common  commercial  practice.  This  example  illustrates  the  challenge 
of  using  market  research  creatively  to  bring  best  commercial  practice  into  a  federal  setting. 
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PPT  and  Best  Value  Prevailed  in 
Source  Selection 
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In  the  chart  above,  we  record  the  reported  source  selection  methodology  used  for  each 
contract,  (This  chart  shows  24  contracts  because  of  a  double  count  for  the  refuse  contract 
and  the  custodial  contract  at  one  base.  In  these  cases,  the  contracting  office  said  that  the 
contracts  were  ‘Tart  12  and  PPT”  contracts.^^)  The  descriptions  of  two  “best  value”^"* 
contracts  in  our  interviews  sounded  more  like  PPT"^  contracts;  however,  this  term  was 
not  used  in  the  interview,  and  there  was  no  information  on  the  federal  government 
electronic  posting  system  (formerly  called  EPS;  now  called  FedBizOpps)  that  was  more 
specific. 


Part  12,  ‘‘Acquisition  of  Commercial  Items,’"  prescribes  “procedures  unique  to  the  acquisition  of 
commercial  items”  and  establishes  “acquisition  policies  more  closely  resembling  those  of  the  commercial 
marketplace.”  These  policies  include  the  recommendation  to  use  past-performance  information  in 
evaluating  proposals,  the  requirement  to  use  FFP  or  FFP  with  economic  adjustment  contracts,  and  the 
requirement  to  rely  in  most  cases  on  a  contractor’s  quality  assurance  systems  as  a  substitute  for  government 
inspection.  Personnel  at  several  bases  mentioned  FAR  Part  12  as  a  component  of  their  acquisition  process 
for  grounds  maintenance,  custodial  services,  and  refuse  collection,  apparently  because  these  services  can 
satisfy  the  definition  of  “commercial  items”  given  in  FAR  Part  2.101. 

^  FAR  Part  15.101  says,  “An  agency  can  obtain  best  value  in  negotiated  acquisitions  by  using  any  one  or  a 
combination  of  source  selection  approaches.”  The  two  primary  approaches  mentioned  as  part  of  the  “best 
value  continuum”  are  “tradeoff”  processes  and  the  Lowest  Price  Technically  Acceptable  (LPTA)  process. 

A  tradeoff  process  permits  tradeoffs  among  cost  or  price  and  noncost  factors  and  allows  the  government  to 
accept  other  than  the  lowest-priced  proposal  as  long  as  the  perceived  benefits  of  the  higher-priced  proposal 
merit  the  additional  cost.  When  a  tradeoff  process  is  used,  the  solicitation  must  clearly  state  the  evaluation 
factors  that  will  be  used  in  the  competition  and  their  relative  importance,  and  it  must  also  state  whether  the 
combined  nonprice  factors  are  significantly  more  important  than,  approximately  equal  to,  or  significantly 
less  important  tlian  price.  The  LPTA  process  (which  is  a  best  value  process  by  definition  in  the  FAR)  is 
used  when  best  value  is  expected  to  result  fi-om  awarding  the  contract  to  the  technically  acceptable  proposal 
with  the  lowest  price.  This  process  does  not  allow  tradeoffs.  None  of  the  contracts  discussed  in  this  study 
used  the  LPTA  approach. 

According  to  Headquarters  Air  Mobility  Command’s  Performance-Price  Tradeoff  Guide,  “PPT  is  a 
simplified  best  value  source  selection  strategy  that  permits  a  tradeoff  between  price  and  performance  in 
reaching  the  award  decision.” 
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People  interviewed  for  this  study  were  enthusiastic  about  best  value  approaches  in 
general  and  PPT  in  particular?*  AMC  has  clearly  encouraged  the  PPT  approach;  70 
percent  of  the  PPT  contracts  were  in  AMC;  and  seven  of  the  eleven  AMC  contracts  used 
this  source  selection  process. 

A  typical  remark  was  that  gathering  past-performance  information  in  the  PPT  process 
was  the  best  way  to  get  the  best  value  for  the  government.  One  interviewee  said,  “[It’s] 
the  best  thing  we’ve  done  in  a  year  and  a  half.”  Another  commented,  “The  [functional] 
customer  loves  the  change  to  PPT. .  .the  happiest  thing  [was  when]  they  told  us,  ‘Low  bid 
doesn’t  win.’”  The  collection  of  past-performance  data  was  also  a  key  incentive  for 
contractors  to  satisfy  their  customers. 

Although  many  of  the  people  we  interviewed  noted  a  push  for  PPT,  liked  how  much  it 
simplifies  the  selection  process,  and  indicated  that  they  wished  they  had  known  about  it 
sooner,  personnel  at  one  AMC  base  expressed  a  preference  for  a  “full”  best  value 
selection  process.  This  group  felt  that  the  PPT  process  can  overlook  too  many  things  and 
that  a  more  complicated  approach  (e.g.,  one  that  includes  comparing  technical  proposals) 
“gets  the  government  more  for  its  money”  and  is  worth  the  additional  work.  A  past- 
performance  assessment  tells  you  how  well  an  offeror  has  performed  against  promises 
made  in  the  past  and  whether  or  not  those  promises  involved  creative  techniques.  Close 
attention  to  the  technical  proposal  tells  you  how  creative  the  offeror  can  be  when 
considering  currently  available  capabilities.  The  two  examine  distinctly  different  aspects 
of  an  offeror’s  desirability.  Positive  scores  on  both  are  important  to  getting  the  best 
performance  for  the  money. 

We  asked  people  if  they  felt  that  using  innovative  acquisition  methods  attracted  a  new 
type  of  vendor.  Only  one  contracting  officer  thought  this  was  the  case.  At  one  base,  30 
vendors  showed  initial  interest  in  a  contract  by  attending  an  open  house;  16  eventually 
made  offers,  and  the  contracting  officer  at  this  base  thought  that  having  a  shorter  SOW 
and  using  commercial  standards  might  have  had  something  to  do  with  the  increased 
interest.  An  interesting  comment  from  personnel  at  one  AMC  base  was  that  the  small 
business  that  won  their  contract  had  teamed  up  with  a  larger  firm — something  they  had 
not  expected.  They  speculated  that  small  businesses  do  not  have  that  much  to  offer  in 
past-performance  data,  but  big  businesses  do,  and  “teaming”  can  give  a  small-business 
offeror  a  boost  because  of  the  larger  business’s  record. 


There  was  apparently  a  minor  controversy  in  1998  about  vAere  PPT  fit  into  the  best  value  continuum. 
FAC  97-02  (dated  September  30, 1997)  implemented  a  rewrite  of  FAR  Part  15,  and  this  rewrite  introduced 
language  that  prohibited  tradeoffs  when  using  LPTA.  At  that  time,  the  Air  Force  considered  PPT  a  form  of 
LPTA.  In  August  1998,  the  Air  Force  issued  Contracting  Policy  Memo  98-C-12,  which  points  out  that 
“with  the  new  FAR  Part  15  definition... PPT  cannot  be  considered  a  form  of  LPTA.  However,  it  is  the  Air 
Force  position  that  PPT  is  still  a  viable  technique  consistent  with  the  spectrum  described  in..  .FAR  15.101.” 
To  solve  the  problem,  AFFAR  Subpart  5315.101-1  was  amended  to  specify  that  PPT  was  a  tradeoff 
process,  not  an  LPTA  process. 
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Questionnaires  and  Contacts  Were 
Sources  of  Past-Performance  Data 

•  Most  used  questionnaires  for  data  collection 

-  MAJCOM  templates  available 

-  Sometimes  modified  for  specific  circumstances 

•  Personal  contacts  used  for  supplemental 
information 

•  Due  dates  for  submission  varied 

•  Some  education  needed  on  importance  of 
providing  good  past-performance  information 


Personnel  at  all  bases  we  contacted  made  use  of  past-performance  data  in  accordance 
with  the  FAR  requirement  to  do  so,  although  there  were  minor  variations  in  approaches 
to  the  data  collection.  Virtually  all  bases  used  questionnaires  that  were  sent  to  previous 
buyers  of  a  vendor’s  services.  Officers  at  one  AETC  base  and  the  ACC  base  said  that 
they  used  a  command  template  or  a  “standard”  questionnaire.  Those  at  another  AETC 
base  created  their  own  questionnaire  despite  the  apparent  availability  of  the  template. 

Some  contracting  officers  gathered  past-performance  information  from  buyers  through 
the  vendors  themselves  by  including  questionnaires  with  the  solicitation;  vendors  could 
then  send  the  questionnaires  to  other  buyers  they  had  worked  with.  Other  officers 
obtained  lists  of  references  from  potential  vendors,  and  the  contracting  office  then  sent 
the  questioimaires  to  these  references.  Follow-up  telephone  calls  were  made  frequently, 
especially  if  there  was  adverse  information.  One  group  of  contract  administrators  also 
pointed  out  that  they  did  not  limit  themselves  to  references  provided  by  the  vendor;  they 
also  used  personal  contacts  to  add  information. 

Past-performance  information  (or  the  reference  list)  was  usually  required  at  the  same  time 
as  the  proposal.  The  AFSPC  contracting  office,  however,  required  past-performance 
information  a  month  before  other  parts  of  the  proposal;  contract  managers  were  pleased 
with  this  approach  because  it  ensured  that  if  they  had  trouble  collecting  past-performance 
information,  this  would  not  delay  proposal  evaluation.  On  the  other  hand,  administrators 
at  one  AMC  base  did  the  same  thing  and  found  that  (1)  they  ended  up  reviewing  the  past 
performance  of  some  vendors  who  never  submitted  proposals,  and  (2)  it  was  unfair  to  ask 
vendors  to  submit  information  in  two  stages  when  those  vendors  did  not  yet  know 
whether  they  would  submit  a  full  proposal. 

As  an  interesting  aside,  interview  results  from  one  base  indicated  that  there  is  some  need 
to  educate  buyers — especially  government  buyers — as  well  as  vendors  about  the 
importance  of  filling  out  past-performance  surveys  carefully.  We  were  told  that  some 
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previous  buyers  “gave  lower  ratings  until  we  told  offerors  to  emphasize  to  previous 
buyers  how  important  the  references  were  so  that  previous  buyers  would  either  take  the 
references  more  seriously  or  give  better  grades  in  the  surveys  they  sent  to  [the 
evaluators].”  Providers  who  had  worked  with  commercial  buyers  in  the  past,  such  as 
airports  or  hospitals,  did  better  on  past-performance  ratings  than  providers  who  had 
worked  only  for  government  buyers.  ITie  Air  Force  informed  “government”  providers  of 
this  finding  at  the  end  of  the  source  selection. 
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Overview 

•  Background 

-  Government  guidance  on  PBSA 

-  Air  Force  emphasis 
>  Goal  of  this  study 

-  Focus  areas  and  interview  sources 

•  Interview  results 

-  Walking  through  the  process 
^  -  Results  of  PBSA  Implementation 

•  Conclusions 


RESULTS  OF  PBSA  IMPLEMENTATION 

We  conducted  these  interviews  primarily  to  collect  information  about  the  services 
acquisition  process  as  discussed  above.  However,  the  interviews  also  yielded  other 
important  findings,  to  which  we  will  now  turn. 
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Study  Found  Satisfaction  with  New 
Contract  Performance 

•  There  was  a  general  belief  that  performance  has 
improved 

•  Objective  evidence  of  improvement  is  tacking 

-  Most  contracts  were  less  than  a  year  old 

-  “Tracking”  metrics  are  lacking 

•  Surprisingly  little  credit  was  given  to  PBSA 

-  “We  were  just  lucky  to  get  a  good  contractor.” 
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Those  we  interviewed  generally  believed  that  the  contracts  they  developed  using  PBSA 
are  working  well.  Indeed,  some  interviewees  unequivocally  approved  of  the  process. 

One  noted  the  difficulty  of  getting  rid  of  an  uncooperative  contractor  when  sealed  bids 
were  used  but  added  that  by  taking  into  account  best  value  instead  of  just  the  lowest  bid, 
the  base  was  able  to  justify  another  vendor.  However,  not  enough  time  has  elapsed  to 
allow  for  an  objective  evaluation  of  changes  in  performance  because  most  of  the 
contracts  we  discussed  in  our  interviews  are  relatively  young — ^i.e.,  in  force  for 
approximately  six  months.  More  disappointingly,  there  is  no  indication  that  bases  are 
making  an  effort  to  formally  compare  the  performance  of  the  current  contractor  with  that 
of  the  previous  contractor.  For  example,  although  contracting  officers  at  some  bases  told 
us  that  complaints  were  down  under  Ae  new  contractor,  they  have  not  analyzed  the 
record  of  complaints  for  the  previous  contractor.  According  to  one  person  we 
interviewed,  “Complaints  are  part  of  the  file,  but  we  don’t  do  formal  tracking  of  them.” 

Other  bases  required  that  the  contractor  keep  metrics  that  can  be  reviewed  by  the  base, 
thus  relying  on  the  contractor  to  track  trends.  Although  this  is  an  effective  approach  for 
managing  the  current  contract,  it  still  does  not  help  the  Air  Force  measure  improvement 
over  previous  contracts. 

We  were  taken  aback  by  the  fact  that,  despite  widespread  satisfaction  with  new  contracts, 
personnel  at  only  two  bases  attributed  the  improvement  they  had  seen  to  the  use  of  new 
contracting  practices  or  to  the  care  they  might  have  taken  in  reviewing  past  performance. 
Common  phrases  we  heard  were  “We’ve  been  lucky  we  got  such  a  good  contractor,” 

“We  may  have  been  lucky,”  and  “We  had  a  lot  of  luck.”  Indeed,  officials  at  both  of  the 
bases  that  credited  new  acquisition  practices  still  maintained  that  they  had  been  lucky! 

We  are  not  sure  if  this  represented  modesty  on  their  part  or  the  assumption  that  since  they 
had  told  us  they  used  PBSA  and  were  satisfied  with  it,  we  should  conclude  that  the 
solicitation  approach  deserved  the  credit. 
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Effect  of  PBS  A  on  Costs  Is  Difficult  to 
Determine 

•  Changes  in  work  scopes  make  it  hard  to  compare  new 
and  old  contracts 

•  Limited  confidence  in  government  estimates  of  costs, 
particularly  in  new  contracting  environment 

•  Effects  on  internal  costs  vary  significantly 

“  tt  always  takes  more  work  the  first  time  you  do  something 

-  Early  steps  of  new  approach  will  generally  take  more  work 

-  Acquisition  team  can  simplify  and  speed  early  work 

-  Demand  for  QAEs  is  falling  more  slowly  than  expected 
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Two  different  kinds  of  costs  are  important  in  estimating  the  effects  of  PBSA  on  service 
acquisition  costs.  The  first  is  the  price  the  Air  Force  pays  for  the  services  delivered.  As 
noted  above,  this  is  usually  a  firm  fixed  price  for  the  contracts  we  examined.  The  second 
is  the  value  of  the  internal  resources  the  Air  Force  commits  to  arranging  and  then 
managing  a  service  acquisition.  The  total  of  these  costs  should  be  taken  into  account  in 
determining  the  effectiveness  of  PBSA  implementation,  but  the  effects  of  a  PBSA 
approach  on  both  types  of  costs  are  difficult  to  assess  for  the  contracts  we  explored. 

The  effect  of  PBSA  practices  on  contract  prices  is  hard  to  assess  for  the  contracts  we 
studied  because  (1)  the  work  scopes  relevant  to  the  contracts  we  examined  changed  with 
the  new  contracts,  and  (2)  the  Air  Force  has  no  simple  way  to  adjust  costs  for  the  changes 
observed  in  work  scopes.  For  example,  the  price  of  one  custodial  contract  was  $45,000 
lower  per  year  than  in  the  previous  contract  even  though  it  actually  included  more 
services  than  did  the  previous  contract,  and  the  ACC  MFH  contract  was  $49,000  cheaper 
annually  despite  an  increase  in  work  requirements.  In  these  two  cases,  the  savings  appear 
to  have  resulted  from  economies  of  scale  realized  through  contract  adjustments  that  were 
a  consequence  of  using  a  best  value  approach.  However,  this  conclusion  is  only 
tentative.  In  most  cases,  we  could  not  clearly  attribute  price  changes  to  a  move  toward 
PBSA. 

One  might  hope  that  the  government  cost  estimates  used  as  a  standard  part  of  an 
acquisition  could  help  overcome  this  problem.  However,  no  one  we  talked  to  was 
comfortable  using  such  estimates  to  explain  changes  in  contract  prices.  In  fact,  when  we 
did  discuss  government  cost  estimates,  we  detected  limited  confidence  in  them.  In  two 
cases,  for  example,  the  winning  offers  were  below  the  government  estimate,  and  the 
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contracting  office  immediately  attributed  this  to  the  government’s  inability  to  make 
accurate  estimates?^ 

Changes  in  “internal”  costs  are  even  more  difficult  to  measure.  We  can  divide  these  into 
“ex  ante”  costs — i.e.,  those  incurred  in  the  preparation  of  a  new  contract — and  “ex  post” 
costs,  or  those  incurred  in  the  management  of  a  contract  once  it  has  been  awarded. 

Discussions  of  these  costs  were  quite  impressionistic.  No  one  had  attempted  to  track  the 
costs  of  the  new  approach  relative  to  the  traditional  approach.  Because  almost  everyone 
involved  in  a  service  acquisition  has  other  simultaneous  duties,  it  is  difficult  in  any  case 
to  sort  out  what  portion  of  the  relevant  players’  time  and  effort  should  be  attributed  to  a 
new  contract.  Indeed,  this  would  be  hard  even  if  the  players  tried  to  track  these  costs  in 
real  time;  thinking  about  such  costs  retrospectively  yielded  a  fairly  wide  range  of 
impressionistic  estimates.  Almost  all  agreed,  however,  that  these  ex  ante  costs  were 
higher  than  they  would  have  been  under  a  traditional  approach. 

Those  we  interviewed  believe  ex  ante  costs  were  higher  with  new  PBSA  contracts  for 
two  reasons.  First,  this  is  a  new  form  of  acquisition.  Personnel  spent  considerable  effort 
preparing  to  do  something  they  had  not  done  before.  Several  emphasized  that  although 
these  costs  were  considerable  for  the  first  effort,  they  should  not  recur  for  subsequent 
efforts.  The  new  costs  appear  to  have  been  more  severe  early  in  the  time  period  we 
examined,  when  available  training  was  limited  and  guidance  was  evolving. 

Second,  the  new  approach  trades  off  greater  investment  in  the  preparation  of  a  contract 
for  lower  administration  costs  and  better  contract  performance  in  the  future.  For 
example,  those  we  interviewed  at  most  bases  told  us  that  market  research  requires 
additional  resources,  and  most  said  it  is  worth  it.  A  smaller  number  said  that  additional 
interaction  with  potential  offerors  added  costs  but  was  generally  worthwhile.  Cross¬ 
functional  teams  made  these  efforts  simpler  than  would  otherwise  have  been  the  case. 
Personnel  at  a  few  bases  strongly  believe  that  their  teaming  arrangements  probably 
reduced  the  time  required  to  prepare  a  new  contract  when  compared  with  a  traditional 
approach.  The  net  effect  of  these  preparation  costs  is  unclear. 

We  expected  that  the  new  focus  on  customer  complaints  would  lead  to  a  reduction  in 
QAE  manpower  or  workload  and  hence  to  a  significant  reduction  in  ex  post  costs. 
However,  the  comments  from  our  interviews  are  ambiguous.  Three  factors  appear  to  be 
important  in  this  context. 

First,  QAEs  play  important  roles  in  the  preparation  of  new  contracts  as  well  as  in  their 
administration.  Even  if  the  new  contracts  are  reducing  demands  for  QAE  administration, 
they  are  increasing  demands  for  QAE  participation  in  new  contract  preparation,  as  noted 
above.  The  net  result  is  that  the  QAE  workload  has  not  dropped  at  many  bases  and  is  not 
expected  to  do  so  imtil  the  development  of  new  contracts  becomes  more  routine,  thereby 
reducing  the  demand  for  QAEs  in  the  preparation  of  new  contracts. 


^^The  changing  acquisition  environment  makes  government  cost  estimation  even  more  difiQcult  than  was 
the  case  in  the  past.  For  example,  the  Defense  Contract  Audit  Agency  has  found  that  because  its  standard 
cost-auditing  tools  are  based  on  historical  cost  data,  these  tools  cannot  be  used  to  anticipate  the  kinds  of 
cost  savings  that  innovative  acquisition  can  generate. 
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Second,  few  bases  are  prepared  to  transition  immediately  to  the  new  approach  to 
administering  contracts.  Formal  inspections  remain  important  in  a  variety  of  critical 
activities.  Moreover,  personnel  who  at  first  claimed  to  rely  100  percent  on  customer 
complaint  for  quality  assurance  indicated,  upon  further  discussion,  that  the  QAEs 
maintain  an  independent  presence  overseeing  execution  and  ensuring  that  (1)  contractors 
understand  this  new  approach,  and  (2)  the  new  approach  in  fact  works  as  well  as 
advertised.  These  QAEs  typically  move  from  a  formal  sampling  approach  to  a  less 
formal  form  of  inspection.  If  the  new  approach  works  as  well  as  anticipated,  these  bases 
expect  to  phase  out  such  inspections  and  the  demand  for  QAEs  associated  with  them. 

Third,  one  or  two  bases  have  made  the  transition  away  from  formal  inspections  to 
customer  complaints  a  formal  component  of  the  way  they  motivate  the  contractor  to 
perform.  The  better  performance  is,  the  less  formal  oversight  the  base  will  maintain.  If 
performance  improves  as  expected,  demand  for  QAEs  will  fall.  If  not,  these  bases  want 
to  reserve  the  capability  to  use  QAEs  more  aggressively. 

The  net  effect  of  these  considerations  is  that  few  bases  have  reduced  their  demand  for 
QAEs.  The  workload  for  QAEs  may  already  have  fallen  at  some  bases,  but  these  bases 
have  not  yet  shifted  these  QAEs  to  other  duties.  At  one  of  the  few  bases  that  had  a 
decrease  in  QA  personnel,  the  loss  of  personnel  came  first,  and  this  reduction  was  one 
reason  (in  addition  to  API  63-124)  the  QASP  was  changed.  Finally,  contract  managers  at 
one  base  pointed  out  that  some  QA  personnel  are  not  excited  about  “writing  themselves 
out  of  a  job”  by  changing  from  inspections  to  customer  complaints  as  a  way  of 
monitoring  contract  performance.  Actual  experience  with  the  demand  for  QAEs 
obviously  bears  close  watching  in  the  future. 
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Attitudes  Toward  PBSA  Are  Generally 
Positive 

•  Experiences  of  most  players  have  been  good 

-  Customers:  Pleasantly  surprised.  Complaints  are  down. 
-Contracting:  Happy.  It  works  the  way  it  should. 

-  Functionals:  Mixed,  is  the  loss  of  control  acceptable? 

-  Contractors:  Happy.  Gives  them  more  freedom. 

•  Players  have  a  mixed  reaction  to  AFI  63-124 

-  Flexibility  creates  new  opportunities  to  improve  contracts 

-  Ambiguity  forces  reliance  on  other  guidance 

•  Experienced  players  emphasize  that  FAR  has  always 
allowed  everything  that  is  happening  today 
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Those  we  interviewed  generally  had  positive  assessments  of  their  experience  with  PBSA. 
Most  reported  that  customer  complaints  had  decreased;  contracting  officers  were  pleased 
with  their  success  in  using  market  research  and  writing  shorter  SOWs;  and  we  were  told 
that  contractors  appreciated  the  greater  independence  they  enjoyed.  Some  functional 
representatives  had  mixed  feelings,  however.  While  they  recognized  increases  in 
customer  satisfaction,  they  sometimes  wondered  if  the  elimination  of  “how  to”  language 
in  SOWs  made  it  more  difficult  for  them  to  control  contractor  performance. 

The  positive  performance  results  mentioned  earlier  were  rarely  attributed  directly  to  the 
use  of  performance-based  contracts,  better  QASPs,  or  formal  incentives  but  were  usually 
credited  instead  to  other  elements  of  the  PBSA  approach.  Typical  comments  about 
successes  were  as  follows: 

“The  low  bid  process  is  antiquated.  The  [best  value]  process  gives  a  better 

product.” 

“If  you  can’t  get  best  value  from  an  offeror,  don’t  go  any  further.” 

“Past  performance  is  the  central  key  to  creating  an  incentive  to  perform.” 

“Contractors  like  [PBSA]... not  so  much  looking  over  [their]  shoulder.” 

Given  that  most  of  these  contracts  were  solicited  when  AFI  63-124  was  approximately 
six  months  old,  it  is  no  surprise  that  there  was  a  wide  range  of  opinion  on  the 
effectiveness  of  that  document.  Personnel  at  one  base  liked  the  flexibility  of  the 
instruction  but  thought  that  some  areas  could  be  more  specific.  However,  they  had  no 
specific  suggestions  on  what  they  would  change.  Officials  at  another  base  simply  said, 
“We  don’t  like  it.  It’s  okay  for  simple  services,  but  it  can’t  deal  with  complex  services  or 
a  large  scope  of  work.”  Since  many  had  applied  the  instruction  when  it  was  new,  there 
were  no  examples  from  which  they  could  borrow  when  they  wrote  their  contracts.  In  the 
words  of  one  person,  this  could  make  for  “a  hellish  time.” 
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Another  complication — again  no  doubt  a  result  of  AFI  63-124’s  newness — was 
occasional  disagreement  with  headquarters  about  what  counted  as  PBSA.  Some 
contracting  officers  mentioned  several  iterations  with  headquarters  contracting  staff  as 
they  refined  their  SOWs  so  that  all  could  agree  that  it  was  performance  based. 

Personnel  at  two  bases  seemed  to  wonder  what  all  of  the  fuss  was  about.  One  said, 
“PBSA  has  been  in  AFM  64-108  [the  Air  Force  manual  on  service  contracts  that  AFI  63- 
124  replaced]  for  years.  It  was  not  really  a  new  approach.”  Another  remarked,  “It 
[PBSA]  has  always  been  there,  but  not  utilized.”  Personnel  who  made  these  comments 
were  quite  experienced,  and  well  informed  about  the  rich  capabilities  available  in  the 
FARs  if  a  contracting  officer  wanted — and  knew  how — to  use  them. 

Perhaps  that  last  remark  is  key:  The  potential  for  PBSA  there  before,  but  people  who 
were  accustomed  to  doing  things  the  “old”  way  did  need  the  push  that  AFI  63-124 
provided  to  encourage  them  to  apply  it. 
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Overview 

*  Background 

-  Government  guidance  on  PBSA 

-  Air  Force  emphasis 

-  Goal  of  this  study 

-  Focus  areas  and  interview  sources 

•  Interview  results 

-  Walking  through  the  process 

>  Results  of  PBSA  Implementation 

Conclusions 


CONCLUSIONS 

We  conclude  with  a  summaiy  of  lessons  learned. 

Although  the  newness  of  PBSA  and  the  lack  of  performance  metrics  make  it  impossible 
for  us  to  objectively  evaluate  the  effectiveness  of  PBSA  implementation  in  the  Air  Force, 
we  were  able  to  identify  policies  and  practices  that  bases  were  pleased  with  and  found 
helpful.  We  also  identified  areas  that  appear  to  require  improvement. 
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Three  Practices  Seem  to  Work  Well 


•  Teamwork 

-  Promotes  “buy-in”  of  PBSA  methods 

-  Promotes  mutual  understanding  between  contracting 
office  and  functional  customer 

•  Market  research 

-  Facilitates  application  of  commercial  standards 

•  Past-performance  evaluation 

-  Gets  the  Air  Force  better  value 

-  Provides  incentive  for  vendors 
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People  in  the  Air  Force  are  applying  PBSA  and  are  generally  pleased  with  the  results. 

One  contracting  officer  said  this  approach  to  acquiring  services  was  “comforting.”  He 
liked  being  able  to  apply  the  same  business  sense  he  uses  in  his  personal  decisions  to 
contracts  for  services  to  the  Air  Force.  Among  the  practices  that  seem  to  work  well  are 
teamwork,  market  research,  and  using  past-performance  information. 

Teamwork  is  important  for  at  least  two  reasons.  First,  PBSA  practices  (and  the  flexibility 
allowed  by  API  63-124)  are  still  new,  and  it  is  important  to  get  people  “on  board”  to 
accept  the  need  for  change.  In  some  cases,  it  was  only  through  the  education  provided  by 
BRAG  discussions  or  an  analogous  cross-functional  team  that  contracting  personnel  were 
able  to  convince  functionals  of  the  effectiveness  of  the  PBSA  approach.  Second, 
cooperation  in  developing  the  SOW  helps  the  contracting  office  understand  the  real  needs 
of  the  functional  customer,  and  the  functional  customer  understand  the  contracting 
office’s  requirements  for  writing  a  performance-based  contract.  Those  who  believed  that 
working  together  helped  meet  the  needs  of  the  functional  customer  while  also  giving  the 
Air  Force  better  value  were  most  pleased  with  the  PBSA  approach. 

Market  research  is  regarded  positively  by  those  we  talked  to  in  the  field,  who  seem 
pleased  with  the  way  it  helped  them  apply  commercial  standards  to  their  SOWs.  Those 
who  were  the  first  in  their  commands  to  use  API  63-124 — i.e.,  those  who  felt  they  were 
exploring  unknown  territory  without  a  map — ^took  the  initiative  to  do  the  market  research 
on  their  own.  Although  many  mentioned  that  market  research  was  time-consuming,  none 
said  that  it  was  particularly  difficult. 

Everyone  was  pleased  with  the  use  of  past-performance  information  in  evaluating 
competitors  despite  the  added  workload  it  required.  They  asserted  that  the  Air  Force  gets 
better  value,  and  the  knowledge  that  past-performance  information  is  used  provides 
incentive  for  vendors  to  do  well  in  order  to  get  new  business  in  the  future. 

This  list  does  not  include  items  that  are  unique  to  performance-based  contracts,  such  as 
SOWs  that  tell  what  is  desired  instead  of  how  to  do  it,  better  QASPs,  and  formal 
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incentive  programs.  As  noted  earlier,  those  we  interviewed  rarely  attributed  improved 
performance  directly  to  these  elements  of  PBS  A.  However,  they  did  have  positive 
attitudes  toward  these  practices — especially  the  new  approach  to  writing  SOWs.  Since 
these  contracts  are  relatively  new,  it  may  be  too  early  to  tell  how  performance  will 
change  as  a  result  of  the  use  of  performance-based  contracts,  and  objectively  measuring 
any  changes  may  thus  be  difficult. 
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Training,  AFis,  and  Metrics  Need 
improvement 

•  Training 

-  Better  understanding  of  commercial  practices 

-  Hands-on  exploration  of  PBSA  with  case  studies 

-  Access  to  training 

-  Howto  get  up-to-date  information 

•  Adequacy  and  understanding  of  APIs,  PARs 

-  Better  examples  in  AFIs  needed 

-  Clearer  interpretation  of  PARs  necessary 

•  Use  of  outcome-oriented  metrics 

-  Relate  PBSA  to  the  mission  of  the  Air  Force 

-  Collect  data  that  can  be  used  to  evaluate  PBSA 

RAND  PnfKiAlitFORCl _  27 


Everyone  we  interviewed  agreed  that  they  had  not  received  enough  formal  training  on 
PBSA.  Many  felt  they  had  received  no  meaningful  training  at  all.  Others  felt  that  the 
basic  awareness  training  they  had  received  was  too  introductory  and  general  to  be  useful. 
In  effect,  this  training  (perhaps  ironically)  told  them  what  PBSA  was  without  telling  them 
how  to  do  it.  If  the  Air  Force  wants  to  fundamentally  change  the  way  it  purchases 
services,  Air  Force  training  should  provide  concrete  details  on  how  to  implement  the  new 
practices. 

To  some  extent,  this  is  more  than  a  training  problem.  Before  the  Air  Force  can  design  a 
course,  it  must  develop  the  material  on  "‘how”  that  goes  into  the  course.  The  most  critical 
comment  came  from  someone  who  pointed  out  that  this  was  a  problem  at  the  Air  Force 
headquarters  level,  because  a  decision  was  made  to  require  the  process  without 
determining  what  was  needed.  However,  this  person  was  also  sympathetic.  The  critic 
noted  that  the  Air  Force  is  early  in  the  transition,  and  the  “experts”  are  learning  along 
with  everyone  else.  This  perspective  suggests  that  the  Air  Force  should  take  advantage 
of  that  learning  and  incorporate  it  into  training  for  use  in  the  future. 

It  appears  that  this  is  beginning  to  happen  to  some  extent.  One  AETC  contracting  officer 
had  just  returned  from  a  two- week  training  course  at  Lackland  Air  Force  Base  and  found 
it  excellent.  “We  now  know  how  to  do  this  and  can  do  better  in  the  future,”  he  said. 

ACC  personnel  also  praised  an  AMC  course  they  had  attended  that  had  educated  them  on 
applying  PBSA  to  military  family  housing. 

The  desire  for  better  training  had  three  themes: 

1.  Exposure  to  specific,  concrete  examples  of  relevant  commercial  practices.  One 

contract  manager  noted  that  his  market  research  had  revealed  that  commercial  firms 
generally  do  not  buy  food  services  in  the  same  way  that  the  Air  Force  does. 
Commercial  firms  buy  not  only  labor  services,  but  also  all  the  assets  and  skills 
relevant  to  providing  restaurant  services.  Better  understandmg  of  this  insight  could 
change  the  Air  Force’s  approach  toward  acquiring  food  services.  Another  noted  the 
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importance  of  practical  new  skills  for  this  new  acquisition  environment.  For 
example,  in  a  PBSA  setting,  commercial  sellers  expect  a  much  higher  degree  of 
courtesy  and  diplomacy  than  Air  Force  QAEs  have  been  trained  to  employ  in  their 
traditional  roles. 

2.  Practical  training  on  the  new  acquisition  process.  Some  of  those  we  interviewed 
recommended  a  course  that  provides  a  step-by-step  approach  to  the  process  with 
suggestions  on  whom  to  involve  at  each  step  and  how  much  effort  and  time  to  expect 
each  step  to  require.^*  Put  another  way,  they  asked  for  clearer  guidance  that  would 
walk  them  through  each  element  of  a  performance-based  acquisition.  Even  if  PBSA 
does  not  lend  itself  to  a  standard  template  for  a  SOW,  a  standard  template  can 
certainly  be  developed  for  the  process  a  base  uses  to  create  its  new  SOW.  Several 
people  stated  that  case  studies  of  successfully  completed  acquisitions  would  help 
them  better  understand  what  to  expect.  Simply  reading  such  case  studies  would  help; 
role  playing  or  gaming  such  cases  could  add  a  useful  aspect  to  training. 

3.  Access  to  training.  One  person  suggested  that  more  training  slots  were  needed  to  get 
experience  levels  up  quickly.  Another  claimed  that  PBSA  was  not  mentioned  in  basic 
introductory  technical  training  courses  for  contracting.  A  third  lamented  the  loss  of 
experience  in  the  business  because  of  shorter  rotations — ^which  creates  the  need  for 
ongoing,  rather  than  one-time,  training  opportunities.  It  is  important  to  note  that 
while  contracting  personnel  are  receiving  some  measure  of  training  on  PBSA,  many 
functional  personnel  are  not.  This  is  particularly  troubling  for  PBSA,  since  it  requires 
teamwork  to  function  effectively. 

The  ability  of  personnel  at  these  bases  to  implement  PBSA — as  well  as  their  satisfaction 
with  their  implementation — is  all  the  more  remarkable  given  this  training  experience  to 
date.  We  were  impressed  by  the  initiative  those  we  interviewed  had  shown  in  finding 
their  own  way  to  the  information  needed  to  complete  their  (typically)  first  performance- 
based  services  acquisition.  However,  there  was  some  frustration  with  the  need  to  search 
for  basic  information  and  the  sometimes  outdated  or  conflicting  guidance  that  appeared 
on  official  websites.  This  could  be  a  continuing  problem.  In  our  research,  we  have  found 
it  difficult  to  keep  up  with  the  proliferation  of  information  on  this  topic,  and  the  number 
of  “central”  websites  seems  to  be  increasing.  Perhaps  part  of  training  development 
should  be  better  guidance  on  where  to  find  information. 

Personnel  at  two  bases  brought  up  the  idea  of  sharing  past-performance  information,  or 
making  information  on  contractors  available  at  a  central  website.  This  idea  remains 
controversial.  For  one  thing,  this  is  not  a  practice  that  the  commercial  world  has  adopted. 
For  another,  it  will  be  time-consuming  and  costly  to  develop  such  a  data  source,  and  it 
will  be  difficult  to  ensure  the  reliability  of  the  information. 


^Qne  training  aid  mentioned  by  more  than  one  base  was  the  Headquarters  Air  Mobility  Command’s 
Performance-Price  Tradeoff  (PPT)  Guide.  SAF/AQ  has  worked  to  make  training  more  available  on  its 
website.  A  series  of  several  briefings  is  currently  available,  with  topics  such  as  What  is  PBSC?  as  well  as 
Writing  SOWs,  Competitive  Sourcing,  and  PBSC  Scenarios.  According  to  SAF/AQCO,  these  on-line 
courses  are  being  well  received. 
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AFI  63-124  is  a  work  in  progress;  a  rewrite  is  currently  under  way.  As  we  have  seen, 
some  personnel  liked  the  vagueness  of  the  first  version;  others  wanted  more  guidance. 

We  believe  that  even  those  who  liked  the  instruction’s  vagueness  would  be  happy  to  see 
more  guidance  such  as  that  addressed  above.  For  example,  the  revised  instruction  could 
provide  more  examples  of  performance-based  SOWs,  including  more  samples  of 
performance  objectives  and  performance  thresholds  from  a  variety  of  services. 

Some  federal  legislation  that  places  restrictions  on  wage  rates  and  competition — such  as 
the  Service  Contract  Act  and  the  Small  Business  Act — is  seen  as  getting  in  the  way  of 
implementing  PBSA.  “How  can  we  act  commercial  when  we’re  hampered  by  these 
laws?”  said  one  interviewee.  “Competition  requirements,  set  minimum  wage  rates... hurt 
us.  [The]  emphasis  is  on  commercial  practices,  but  our  hands  are  tied.”  The  writers  of 
the  AETC  refuse  contract  felt  that  it  was  difficult  to  build  an  incentive  scheme  when  a 
contract  is  limited  by  FAR  Part  12  to  variations  on  an  FFP  arrangement. 

It  is  unlikely  that  such  regulatory  limitations  will  change,  but  Air  Force  personnel  would 
benefit  from  a  better  imderstanding  of  which  limitations  are  really  binding  and  what 
options  are  available  in  the  face  of  these  limitations.  PBSA  suggests  that  the  Air  Force 
should  approach  standing  limitations  in  a  new  way.  Rather  than  taking  a  conservative 
stance  or  assuming  that  nothing  is  allowed  unless  it  is  explicitly  sanctioned,  PBSA 
endorses  a  creative  approach  that  seeks  a  way  around  regulatory  obstacles.  Everything  is 
allowed  unless  it  is  explicitly  prohibited,  and  even  then,  creative  opportunities  exist  to 
frame  prohibitions  in  more  favorable  ways.  Many  of  the  personnel  we  talked  to  embody 
this  spirit — ^those  who  are  using  award-term  contracts,  for  example.  Less  motivated 
personnel  will  need  more  encouragement  than  this  first  wave  of  innovators  has  required. 
Training  could  profitably  emphasize  the  creative  nature  of  PBSA  and  illustrate  it  with 
concrete  examples  of  what  can  be  done  and  resulting  performance  improvements. 

The  Air  Force  PBSA  program  does  not  mandate  formal  tracking  of  performance  and  cost 
to  evaluate  how  PBSA  is  affecting  outcomes  relevant  to  ultimate  Air  Force  customers, 
such  as  war  fighters  and  military  families.  Thus,  the  fact  that  no  base  we  examined  is 
tracking  such  metrics  does  not  mean  that  Air  Force  PBSA  is  not  working  as  intended. 

But  it  is  an  important  aspect  of  current  PBSA  in  the  Air  Force  that  “doesn’t  work”  in  the 
Air  Force’s  best  interests.  In  failing  to  track  metrics,  the  Air  Force  is  missing  a  chance  to 
document  the  actual  outcomes  of  its  early  experience  with  PBSA  and  to  use  such 
documentation  to  (1)  promote  further  application  of  PBSA  throughout  the  Air  Force,  and 
(2)  continually  improve  the  application  of  PBSA  as  the  Air  Force  learns  from  its 
accumulating  experience.  The  adoption  of  performance-based  contracting  methods  is 
ultimately  a  way  to  help  the  Air  Force  accomplish  its  mission  more  efficiently  and  to 
obtain  the  best  value  for  the  American  taxpayer.  The  support  of  leadership  at  the  base 
level  is  critical  to  the  adoption  of  PBSA  methods,  but  without  data  to  show  that  PBSA  is 
helping  them  do  their  job  better,  these  leaders  will  be  hard  to  win  over. 
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This  is  not  a  simple  problem.  Previous  RAND  research  indicates  that  the  best  firms  in 
the  commercial  sector  are  having  a  difficult  time  measuring  the  benefits  of  PBSA,  too — 
even  though  they  share  the  general  feeling  that  it  improves  performance.^’ 

In  our  interviews,  we  detected  little  interest  in  developing  such  measures  or  in  collecting 
data  that  might  be  useful  for  studying  this  in  the  futme.  Personnel  in  the  field  face  many 
competing  demands  on  their  time,  and  they  do  not  see  the  value  of  such  information  to 
their  immediate  efforts.  They  will  not  collect  it  as  aggressively  as  they  have  pursued 
other  aspects  of  PBSA  unless  they  understand  how  the  Air  Force  as  a  whole  can  benefit 
from  such  information. 

This  summary  of  what  needs  to  be  improved  should  not  detract  from  the  “bottom  line”  of 
our  interview  results:  In  its  opening  round  of  performance-based  contracts,  motivated 
people  in  the  Air  Force  services  acquisition  business  were  able,  despite  limited  and 
sometimes  contradictory  information,  to  implement  PBSA  practices  in  a  variety  of 
services.  Most  are  pleased  with  the  results,  even  if  objective  measures  of  their  success 
are  not  yet  available. 


^®See  Laura  H.  Baldwin,  Frank  Canun,  and  Nancy  Y.  Moore,  Strategic  Sourcing:  Measuring  and 
Managing  Performance^  DB-287-AF,  Santa  Monica:  RAND,  2000. 
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Appendix  A 

LESSONS  FOR  FUTURE  SERVICES  ACQUISITIONS 


Personnel  we  talked  to  at  each  base  emphasized  different  lessons  important  to  their  own 
recent  experience,  and  the  list  below  draws  on  their  collective  experiences.  The  lessons 
offered  are  consistent  with  (1)  specific  lessons  identified  by  those  we  interviewed,  (2) 
what  we  observed  in  the  acquisitions  we  examined  in  this  project,  and  (3)  lessons 
identified  in  related  analyses  that  examined  similar  services  acquisitions  elsewhere. 

Successful  implementation  of  PBSA  will  be  more  likely  and  more  complete  if  Air  Force 
bases  initiating  new  service  acquisitions  do  the  following: 


GETTING  STARTED 

•  Get  all  Air  Force  stakeholders  to  buy  into  using  the  new  approach  to  services 

acquisition.  Three  factors  are  important: 

•  Get  the  support  of  the  base  leadership.  Do  this  by  (1)  relating  the  proposed 
changes  to  factors  that  the  base  leadership  cares  about,  and  (2)  keeping  the 
leadership  informed  about  how  the  unfolding  acquisition  affects  the  factors 
identified.  The  leadership’s  time  is  precious.  When  you  have  its  attention,  focus 
on  how  a  new  approach  affects  the  base,  not  specific  functional  metrics. 

•  Train  all  relevant  Air  Force  players,  from  the  base  leadership  through  the 
functional  customer  communities  to  the  contracting  office  and  any  other 
fimctionals  that  will  be  involved  (e.g.,  financial  management,  small  business, 
competition  advocate).  Demonstrate  why  the  new  approach  is  important  to  the  Air 
Force.  Give  them  the  skills  they  need  to  succeed. 

•  Use  teams  to  maintain  the  involvement  and  buy-in  of  all  relevant  commrmities. 
Tailor  teams  to  the  acquisition  at  hand;  many  variations  are  possible.  Schedule 
periodic  team-based  reviews  of  contract  performance,  and  include  base  leadership 
in  incentive  reward  decisions. 

•  Emphasize  performance  relevant  to  the  Air  Force  as  a  whole  over  compliance  with 

specific  new  acquisition  policies. 

•  Mere  compliance  with  new  Air  Force  guidance  does  not  guarantee  a  significant 
increase  in  military  capability  or  quality  of  life  on  a  base  or  a  reduction  in  total 
ownership  costs  to  the  Air  Force.  In  fact,  simply  learning  to  do  things  in  a  new 
way  can  impose  costs  without  yielding  offsetting  benefits. 

•  Increased  emphasis  on  creative  use  of  the  new  tools  and  capabilities  now  available 
is  more  likely  to  benefit  the  Air  Force  than  a  focus  on  simply  complymg  with  new 
policy. 

•  Develop  a  services  acquisition  plan  in  which  the  elements  of  the  plan  work  together. 

For  example: 
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•  Market  research  and  source  selection  will  both  work  better  if  they  work  together. 
Market  research  helps  frame  an  acquisition  to  attract  the  right  sources;  early 
contact  with  industry  in  source  selection  (e.g.,  requests  for  information,  “industry 
days”)  refines  initial  market  research. 

•  The  flexibility  allowed  in  performance-based  SOWs  and  quality  assurance  plans 
presents  fewer  risks  if  (1)  the  Air  Force  can  choose  a  high-quality,  reliable 
provider  that  does  not  require  detailed  oversight,  and  (2)  the  winning  provider 
knows  that  its  performance  on  this  contract  will  affect  its  success  in  future 
competitions. 

•  Reliance  on  customer  complaint  as  an  element  in  quality  assurance  works  better 
when  the  customers  who  are  allowed  to  complain  clearly  understand  the  terms  of 
the  contract. 

•  Provide  appropriate  training  when  it  is  needed  and,  if  possible,  tailored  to  the 
acquisition  in  question. 

•  Training  shortly  before  an  acquisition  starts  is  helpful  because  (1)  it  is  easiest  to 
apply  new  skills  shortly  after  they  are  learned,  (2)  training  just  before  an 
acquisition  is  easier  to  tailor  to  that  acquisition,  and  (3)  with  continuing  staff 
turnover,  such  training  increases  the  likelihood  that  all  relevant  players  have  the 
awareness  and  skills  they  need. 

•  Provide  awareness  training  to  all  relevant  players. 

•  Provide  more  detailed,  “how-to”  training  tailored  to  the  tasks  each  player  will 
execute  in  the  acquisition.  All  players  need  concrete  training  on  the  basic  steps  in 
a  new  kind  of  acquisition,  the  players  relevant  to  each  step,  their  roles  and 
responsibilities,  and  reasonable  estimates  of  schedule  and  resource  requirements 
for  each  step.  If  players  must  exercise  judgment  about  designing  a  source 
selection  or  statement  of  objectives,  case-based  training  that  supports  such 
decisionmaking  will  help.  If  they  will  be  working  in  a  team  setting,  training  in  a 
team  setting  will  provide  good  preparation.  If  the  training  uses  examples  keyed  to 
the  acquisition  in  question  (e.g.,  A-76,  small-business  set-aside,  a  particular 
service),  application  of  new  skills  learned  will  be  more  direct. 

•  Keep  relevant  players  engaged  in  effective  teams  throughout  the  course  of  the 
acquisition  and  subsequent  contract  execution.  Tailor  membership  and  frequency  of 
meetings  to  the  tasks  at  hand.  The  points  below  offer  more  detail. 

•  Expect  the  new  approach  to  take  more  time  than  the  traditional  approach  the  first  time 
it  is  applied.  Plan  for  this  in  activity  assignments,  staffing  decisions,  and  more  general 
resource  planning.  This  cost  is  likely  to  fall  fairly  rapidly  as  experience  accumulates 
at  a  base. 

•  Formal  and  informal  training  will  absorb  staff  time  and  perhaps  financial  resources 
leading  up  to  and  through  a  new  acquisition. 

•  Staff  members  will  take  longer  to  do  things  the  first  time  they  do  them.  They  will 
make  mistakes  that  must  be  corrected.  They  will  puzzle  over  new  responsibilities. 
Such  inefficiencies  are  investments  in  the  future. 
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•  Coordination  between  bases  and  their  MAJCOM  headquarters  may  take  longer  as 
they  work  out  new  procedures  and  the  specific  application  of  new  policies. 


MARKET  RESEARCH 

•  Use  market  research  to: 

•  Help  frame  requirements  in  ways  that  attract  better  potential  providers  by  allowing 
them  to  do  what  fliey  do  best  in  a  commercial  setting  (e.g.,  buying  food  services 
rather  than  the  services  of  food  servers). 

•  Help  frame  acquisition  methods  in  ways  that  attract  better  potential  providers  by 
reducing  their  participation  costs  and  helping  them  shape  Air  Force  acquisitions  to 
match  their  capabilities  to  Air  Force  needs  (e.g.,  interaction  with  potential 
providers,  pricing  and  payment  policies,  quality  assurance  systems). 

•  Identify  the  best  potential  providers  themselves  and  seek  specific  ways  to  attract 
them  to  competitions. 

•  To  frame  market  research,  look  ahead  through  the  acquisition  and  ask  what  kinds  of 
information  about  commercial  practices  and  players  are  relevant  to  each  step  of  the 
acquisition.  Properly  conducted,  such  market  research  can  shape  every  aspect  of  an 
acquisition. 

•  It  helps  the  Air  Force  recognize  standard  commercial  practices,  which  must  often 
be  carefully  adapted  to  Air  Force  use. 

•  It  helps  the  Air  Force  tailor  individual  acquisitions  to  the  services  in  question  and 
practices  peculiar  to  particular  locales.  Expect  considerable  variation  across 
services  and,  for  some  services,  across  locales. 

•  Involve  all  players  relevant  to  the  issues  identified  above  in  market  research. 
Contracting,  functional  customers,  and  other  communities  have  different  questions  to 
ask  in  market  research.  Encourage  these  communities  to  work  together  so  that  each 
community  can  learn  what  it  needs  to  know  about  commercial  practices  relevant  to  the 
Air  Force. 

•  Do  not  be  fiightened  off  by  the  ambitious  agenda  proposed  above  for  market  research. 
Simpler  versions  can  add  value  with  only  limited  preparation  and  training. 

•  Simple  review  of  potential  providers  in  the  local  Yellow  Pages  can  provide  a  start. 
More  complete  industry  directories  are  easily  accessible.  Potential  providers  are 
generally  open  to  inquiries  relevant  to  Air  Force  market  research. 

•  Local  buyers  of  similar  services  can  be  quite  helpful  in  identifying  relevant 
commercial  practices  and  standards  specific  to  a  service  or  to  the  terms  on  which 
firms  buy  and  sell  the  service.  Such  firms  are  willing  to  spend  more  time  if  the  Air 
Force  brings  information  to  them  that  they  value.  This  will  become  easier  as  the 
Air  Force  accumulates  more  information  that  it  can  share. 
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•  Discussions  on  the  phone  are  generally  easier  to  arrange  and  execute  than  face-to- 
face  meetings  and  are  generally  more  productive  than  asking  a  firm  to  complete  a 
questionnaire  in  writing,  particularly  if  the  firm  is  not  a  potential  provider. 

•  “Industry  days”  can  attract  a  great  deal  of  interest,  but  are  more  successful  if 
carefully  planned  to  convey  and  elicit  specific  pieces  of  information.  Careful 
market  research  can  be  invaluable  in  focusing  such  events. 

•  More  complex  forms  of  market  research — such  as  formal  benchmarking  studies, 
benchmarking  networks,  and  third-party  surveys  beyond  a  locale — can  be 
extremely  useful,  but  they  are  not  necessary.  Air  Force  bases  can  get  started  on 
effective  market  research  without  anything  this  elaborate. 


DEFINING  REQUIREMENTS 

•  Use  requirements  relevant  to  the  base  and  to  the  Air  Force  as  a  whole.  To  the  full 
extent  possible,  identify  the  operational  requirements  relevant  to  a  service  the  Air 
Force  is  acquiring.  Use  these  requirements  to  focus  on  the  value  the  Air  Force  expects 
to  get  fi'om  a  service,  not  the  methods  used  to  provide  that  service.  It  is  often  easier 
for  the  ultimate  Air  Force  customer  who  will  benefit  from  a  service  to  think  this  way 
than  it  is  for  the  functional  customer  who  has  traditionally  defined  how  the  Air  Force 
wants  the  work  done. 

•  Use  market  research  to  propose,  review,  and  refine  requirements: 

•  Can  requirements  tied  to  traditional  Air  Force  functional  guidance  be  reffamed  in  a 
way  that  (1)  reflects  higher-level  goals,  and  (2)  clearly  tells  potential  commercial 
offerors  what  the  Air  Force  really  cares  about? 

•  To  the  extent  that  the  stated  requirements  reflect  detailed  guidance  from  standing 
Air  Force  instructions  and  other  policies,  does  such  guidance  preclude  approaches 
observed  in  the  commercial  sector  that  might  work  in  an  Air  Force  setting?  If  so, 
what  difference  between  the  Air  Force  and  the  commercial  sector  makes  these 
commercial  approaches  inappropriate  to  the  Air  Force?  What  compromises  are 
possible? 

•  To  the  extent  that  MAJCOM  guidance  mandates  requirements  to  ensure  that  the 
Air  Force  seeks  the  minimum  quality  of  service  appropriate  for  government  use, 
do  such  requirements  differ  from  requirements  stated  in  the  commercial  sector?  If 
so,  why?  Is  it  possible  to  show  that  a  higher  level  of  quality  could  promote  the 
interests  of  the  Air  Force  as  a  whole? 


SOURCE  SELECTION 

•  Use  past  performance  as  a  major  criterion  in  any  source  selection. 

•  It  yields  a  distinctly  different,  better  provider  for  the  Air  Force,  even  when  an 
acquisition  attracts  the  same  offerors  that  have  come  to  past  acquisitions  of  the 
same  service. 
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•  Bases  differ  on  the  relative  desirability  of  (1)  a  full  best-value  source  selection,  and 
(2)  a  slinuned-down  “tradeoff”  version  that  focuses  on  past  performance  and  price. 
The  slimmed-down  version  is  easier  to  administer,  but  does  not  yield  important 
information  about  technical  capability  that,  in  some  circumstances,  may  be  worth 
the  extra  effort  it  requires. 

•  To  date,  few  losing  offerors  have  used  past-performance  data  successfully  to 
challenge  the  Air  Force’s  choice  of  a  winner. 

•  Use  a  definition  of  past  performance  appropriate  to  each  acquisition. 

•  Questionnaires  based  on  standard  templates  and  given  to  offerors  to  send  to  then- 
other  customers  are  fairly  easy  to  administer. 

•  Questionnaires  tailored  to  the  factors  relevant  to  a  specific  acquisition  can  give  the 
Air  Force  additional  data  important  to  a  specific  decision. 

•  Direct  Air  Force  communication  with  other  customers  can  clarify  statements  on 
questionnaires  relevant  to  an  Air  Force  decision. 

•  These  observations  suggest  that,  even  if  centrally  maintaining  past-performance 
data  cuts  the  costs  to  the  Air  Force  of  gathering  such  data,  data  tailored  to  a 
particular  acquisition  could  easily  be  worth  the  extra  effort. 

•  Asking  offerors  to  submit  past-performance  data  before  their  proposals  (1)  gives  the 
Air  Force  extra  time  to  process  past-performance  data,  but  (2)  imposes  unnecessary 
costs  on  the  Air  Force  and  offerors  when  offerors  that  have  submitted  past- 
performance  data  decide  not  to  submit  a  proposal.  Bases  differ  on  the  relative  merit  of 
these  two  approaches. 


WRITING  THE  STATEMENT  OF  OBJECTIVES  AND 

QUALITY  ASSURANCE  PLAN 

•  Coordinate  the  requirements,  statement  of  objectives,  and  quality  assurance  plan 
carefully. 

•  Clearly  link  requirements  and  objectives. 

•  Clearly  link  objectives  to  metrics  to  measure  performance,  or  judge  a  proposal 
carefully  on  how  it  makes  this  link. 

•  Verify  that  any  metrics  approved  (1)  can  be  measured  with  data  from  available 
information  systems,  and  (2)  are  likely  to  induce  performance  that  promotes  Air 
Force  goals  as  they  are  reflected  in  the  requirements  stated. 

•  Provide  a  mechanism  for  adjusting  the  metrics  used  to  manage  performance 
without  reopening  the  contract  if  it  is  determined  that  these  metrics  do  not  induce 
the  performance  desired. 

•  To  the  full  extent  possible,  reflect  the  goals  described  above  in  initial  discussions  with 
potential  providers  and  in  the  formal  request  for  proposals.  Help  potential  providers 
imderstand  the  regime  in  which  performance  management  will  occur,  and  work  with 
them  to  design  a  regime  that  (1)  allows  them  to  use  quality  control  information 
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systems  and  procedures  that  they  use  with  commercial  buyers,  and  (2)  meets  the 

particular  procedural  requirements  of  federal  procurement. 

•  Match  appropriate  incentives  to  the  metrics  used  to  manage  performance. 

•  Incentives  that  the  bases  currently  use  include  (1)  simple  firm  fixed  prices, 
adjusted  only  for  selected  cost  changes,  (2)  using  today’s  performance  to  affect 
future  competitions,  (3)  award  fees,  and  (4)  award  terms.  The  Air  Force  has  not 
yet  considered  other  incentive  systems  often  observed  in  commercial  practice. 

•  Incentives  vary  across  services  and  locations.  Some  of  this  variability  appears  to 
be  justified  by  differences  in  the  services  and  the  priorities  of  the  bases  buying 
these  services.  Other  variability  is  likely  to  decrease  as  the  Air  Force  learns  more 
about  the  actual  performance  of  specific  incentives  and  the  compatibility  of 
commercial  practices  with  Air  Force  priorities.  The  development  of  effective 
metrics  to  monitor  performance  changes  will  help  the  Air  Force  evaluate 
incentives  and  plan  for  the  future. 

•  Experience  to  date  suggests  that  an  award  fee  allows  (1)  enough  subjective 
judgment  to  compare  provider  performance  effectively  to  actual  Air  Force  goals, 
and  (2)  enough  procedural  structure  to  limit  provider  objections  to  how  it  is 
administered.  Award  fees  work  best  when  a  fairly  transparent  process  is  used  to 
pick  an  award  level  and  the  provider  is  given  clear  feedback  on  the  priorities  that 
yield  each  award  decision. 

•  Use  appropriate  teams  to  write  the  statement  of  objectives  and  quality  assurance  plan. 

•  The  ultimate  customer  of  a  service,  including  the  base  leadership  as  needed,  can 
provide  input  on  appropriate  requirements  and  objectives  and  how  to  specify  them 
in  any  quality  assurance  plan. 

•  Functional  customers  will  provide  the  quality  assurance  evaluators  who  manage 
performance  and  can  suggest  how  to  frame  a  plan  to  make  the  evaluators  as 
effective  as  possible. 

•  Contracting  officials  know  details  about  the  new  approach  to  services  acquisition 
and  the  requirements  of  federal  procurement  regulations.  This  can  help  shape 
relevant  commercial  practices  to  the  federal  setting. 

•  Other  functionals,  such  as  financial  management,  bring  analogous  skills  and 
perspectives  to  these  activities. 

Teams  comprising  these  stakeholders  (1)  tend  to  reduce  the  time  and  effort  required  to 
coordinate  information  flow  and  decisions  that  involve  more  than  one  player,  (2)  give  all 
players  a  greater  sense  of  ownership,  and  (3)  give  an  Air  Force-wide  perspective  greater 
visibility  relative  to  any  one  player’s  priorities.  Such  teams  will  produce  statements  of 
objectives  and  quality  assurance  plans  that  are  more  likely  to  reflect  Air  Force-wide 
priorities  and  to  induce  a  provider  to  pursue  these  priorities. 
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PERFORMANCE  MANAGEMENT 

•  Adjust  the  role  of  the  quality  assurance  evaluator  to  reflect  the  new  acquisition 
environment.  For  example: 

•  Give  greater  attention  to  collecting  information  on  customer  complaints  than  on 
details  of  the  process  a  provider  uses  to  produce  a  service.  In  particular,  move 
from  sampling  and  surveying  data  about  providers  to  sampling  and  surveying  data 
about  the  satisfaction  of  tiie  actual  customers  who  use  a  purchased  service. 

•  Give  greater  attention  to  working  with  the  actual  customers  who  use  a  purchased 
service  in  order  to  understand  the  factors  important  to  their  satisfaction  and  to  help 
them  imderstand  the  terms  of  any  standing  contract. 

•  Give  greater  attention  to  working  with  providers  to  solve  problems  as  they  arise, 
and  give  less  attention  to  documenting  failure  to  perform  and  calculating 
deductions  from  the  price  justified  by  a  failure  to  perform.  More  generally,  help 
quality  assurance  evaluators  maintain  a  more  positive  and  less  confrontational 
relationship  with  providers. 

•  Be  prepared  to  retain  quality  assurance  evaluators  imtil  (1)  it  is  confirmed  that  the 
success  of  new  acquisition  methods  actually  eliminates  the  Air  Force’s  need  for  such 
evaluators  to  inspect  and  formally  sample  performance  on  a  continuing  basis,  and  (2) 
the  Air  Force  no  longer  needs  evaluators’  input  on  teams  designing  new  acquisitions. 

•  Use  appropriate  teams  to  advise  quality  assurance  evaluators  as  they  continue  to 
monitor  customer  satisfaction  and,  where  appropriate,  provider  performance  more 
directly. 
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Appendix  B 

INTERVIEW  QUESTIONNAIRE  TEMPLATE 


Telephone  interviews  for  this  project  were  structured  around  the  following  questions. 
Although  the  interviews  themselves  were  free-flowing,  information  in  the  raw  transcripts 
was  transcribed  into  this  template. 

1.  What  does  the  new  contract/QASP  look  like?  How  does  it  differ  from  the  old 
contract? 

1.1.  Activity  covered?  Scope?  Change  from  before? 

1 .2.  Date  awarded  or  initiated 

1.3.  Base/option  structure 

1.4.  Type?  Award  fee? 

1.5.  Set-aside? 

1.6.  Value?  Relative  to  previous  contract? 

1.7.  Length  of  SOW?  Change  from  before?  Reasons? 

1.8.  QAplan?  Change  from  before?  Reasons? 

1.9.  Any  special  clauses?  Change  from  before?  Reasons? 

2.  What  led  you  to  make  these  changes?  Direction  from  above? 

3.  How  did  you  prepare  for  the  new  approach,  step  by  step? 

3.1.  Acquisition  planning/BRAG/involvement  of  customer 

3.2.  Market  research 

3.3.  Early  contact:  site  visit,  etc. 

3.4.  Source  selection — qualification,  best  value,  PPT  issues 

3.5.  Development  of  SOW/QASP.  Application  of  commercial  standards. 

4.  What  resources  were  required  to  develop  the  contract?  How  long  did  it  take? 
Compared  to  the  past? 

5.  What  training  did  you  get?  What  training  did/do  you  need? 

6.  How  are  your  PBSCs  working  to  date?  Is  performance  different  from  before? 
Any  ability  to  document  changes  in  quality  of  performance? 

7.  How  well  did  API  64-123  work  for  you?  What  else  did  you  use?  AFCESA 
templates? 

8.  What  would  you  do  differently  if  you  could  do  it  again? 

9.  Any  suggestions  for  the  rest  of  the  Air  Force? 
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Appendix  C 

INTERVIEW  PARTICIPANTS  BY  CONTRACT 


Contract 

Contract  Type 

Interview  Participants 

1 

Base  operations 
support 

Contracting  officer  (civilian) 

Program  manager,  evaluation  team  chief  (military) 

2 

Custodial 

services 

QAE  (civilian) 

Contracting  specialist  (civilian) 

3 

Custodial 

services 

Contracting  specialist  (civilian) 

Functional  manager  (civilian) 

4 

Custodial 

services 

Two  contracting  officers  (one  military,  one  civilian) 

5 

Custodial 

services 

Contracting  office  commander  (military) 

Director  of  business  operations  (civilian) 

Three  others  from  contracting  office  (all  civilians) 

6 

Custodial 

services 

Two  from  contracting  office  (both  civilians) 

Technical  adviser  from  civil  engineering  (civilian) 

7 

Food  services 

Three  from  QAE  (one  military,  two  civilian) 

Contract  administrator  (civilian) 

Contract  manager  (military) 

8 

Food  services 

QAE  (civilian) 

Contracting  specialist  (civilian) 

9 

Grounds 

maintenance 

Contract  administrator  (civilian) 

Business  operations  representative  (civilian) 

10 

Groimds 

maintenance 

Three  from  contracting  office  (all  civilians) 

Three  from  civil  engineering  QAE  (including  the 
program  coordinator;  all  civilians) 

11 

Grounds 

maintenance 

One  QAE  for  the  contract  (military) 

12 

Grounds 

maintenance 

Acting  contracting  chief  (civilian) 

Contract  administrator  (civilian) 

Functional  area  chief  (civilian) 

QAE  (civilian) 
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Interview  Participants  by  Contract  (continued) 


13 

Grounds 

maintenance 

Civil  engineering  (military) 

QAE  (civilian) 

14 

MFH 

Director  of  contract  operations  (civilian) 

Contracting  civil  engineering  acquisitions  (civilian) 

15 

MFH 

Officer  who  awarded  the  contract  (civilian) 

Two  from  contracting  office  (one  military,  one  civilian) 
MFH  representative  (civilian) 

16 

MFH 

Contracting  officer  (civilian) 

Functional  chief  for  MFH  (civilian) 

17 

MFH 

Civil  engineering  (civilian) 

Contracting  office  (civilian) 

18 

Refuse 

Contracting  office  commander  (military) 

Director  of  business  operations  (civilian) 

Three  others  from  contracting  office  (all  civilian) 

19 

Other 

(security) 

Contract  manager 

20 

Other 

(variable  staff 
requirement) 

Contract  specialist  (civilian) 

Functional  representative  (civilian) 

21 

Other 

(communications 

equipment) 

Civil  engineering  (military) 

Contract  specialist  (civilian) 

22 

Other 

(training 

equipment) 

Contract  manager  (military) 
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